P516804.PDF  [Page:  1  of  100] 

Image  Cover  Sheet 


P516804.PDF  [Page:  2  of  100] 


This  page  is  left  blank 


This  page  is  left  blank 


P516804.PDF  [Page:  3  of  100] 


DCIEM  No.  CR-2001-071 


Evaluation  of  Potential  Test  Environments  for 
Assessing  the  Impact  of  Multi-Sensor  Data  Fusion  on 
Command  and  Control  Operations  in 
the  HALIFAX  Class  Frigate 


by: 

Michael  L.  Matthews,  Robert  D.G.  Webb, 
Paul  Stager  and  Roy  Keeble 

Humansysfems,  Incorporated 
111  Farquhar  St.,  2nd  floor 
Guelph,  ON  N1H3N4 
(519)  836  5911 

Project  Manager: 

(519)  836  5911 


PWGSC  Contract  No.  W771 1-7-7404/01 -SV 
Call-Up  No.  7404-21 


On  behalf  of 

DEPARTMENT  OF  NATIONAL  DEFENCE 


DCIEM  Scientific  Authority 
Carol  McCann 
(416)  635-2190 


May  2001 


P516804.PDF  [Page:  4  of  100] 


Abstract 


This  report  reviews  five  training  simulator  facilities  in  the  Canadian  Navy  for  their  suitability  in 
conducting  test  and  evaluation  (T&E)  in  support  of  the  COMDAT  multi-sensor  data  fusion  (MSDF) 
initiative.  The  facilities  are  the  Operations  Room  Team  Trainer  (ORTT),  the  Naval  Combat  Operator 
Trainer  (NCOT),  the  Combat  Systems  Training  Centre  (CSTC)  and  HALIFAX  Class  Ops  Room 
alongside  or  at  sea.  A  set  of  detailed  capabilities  for  conducting  T&E  was  developed  under  the  four 
major  categories  of:  Logistics,  Experimental  Control,  Data  Collection  and  Data  Storage  and 
Analysis.  Each  of  the  potential  facilities  was  reviewed  against  these  categories  and  the  capabilities 
of  each  facility  were  then  assessed  for  their  suitability  for  conducting  the  expected  range  of  T&E 
activities.  It  was  concluded  that  the  NCOT  would  be  the  most  suitable  facility  for  conducting  T&E 
using  single  operators  and  small  teams  performing  simple  operational  functions  such  as  detecting 
and  tracking  radar  contacts.  For  more  complex  functions  in  the  Ops  Room,  including  tasks  involving 
interactions  between  complete  warfare  teams,  interactions  between  warfare  directors  and  the  ORO, 
and  between  the  ORO,  CO  and  the  entire  Ops  Room  team,  the  ORTT  represents  the  appropriate 
environment  for  conducting  T&E  activities.  A  range  of  scenarios  exists  for  the  ORTT  and  for 
NCOT,  which  may  be  adaptable  for  T&E  purposes.  Severe  constraints  on  the  availability  of  the 
ORTT  (and  possibly  the  NCOT)  and  the  associated  personnel  infrastructure  were  noted,  due  to 
existing  training  demand  from  the  Navy.  It  is  recommended  that  a  simple  proof  of  concept  trial  for 
assessing  the  suitability  NCOT  for  T&E  be  conducted  as  soon  as  possible. 


Resume 


Le  present  rapport  porte  sur  la  pertinence  d’utiliser  les  cinq  simulateurs  d'entrainement  de  la  Marine 
canadienne  pour  effectuer  les  essais  et  revaluation  (E  et  E)  de  l’initiative  de  fusion  de  donnees  des 
multicapteurs  COMDAT  (MSDF).  Les  installations  comprennent  le  simulateur  pour  l’equipe  de  la 
salle  des  operations  (SESO),  le  simulateur  d'operations  de  combat  naval  (NCOT),  le  centre 
destruction  des  systemes  de  combat  (CSTC)  et  la  salle  des  operations  de  classe  HALIFAX,  a 
proximity  ou  en  mer.  Une  serie  de  capacites  detaillees  permettant  d’effectuer  les  E  et  E  a  ete 
identifiee  en  fonction  des  quatre  grandes  categories  que  sont  la  logistique,  le  controle  experimental, 
la  collecte  des  donnees  et  l’archivage  et  1’ analyse  des  donnees.  Chaque  installation  a  ete  etudiee  a  la 
lumiere  de  ces  categories  et  les  capacites  de  chaque  installation  ont  ensuite  ete  evaluees  en  ce  qui  a 
trait  a  leur  pertinence  pour  effectuer  toute  la  gamme  prevue  des  activites  d’E  et  E.  II  apparait  que  le 
NCOT  serait  1’ installation  la  plus  appropriee  pour  effectuer  les  E  et  E  tant  avec  des  operateurs  seuls 
qu’avec  des  equipes  restreintes  pour  les  fonctions  operationnelles  simples  telles  que  la  detection  ou 
les  poursuites  de  contacts  radars.  Pour  les  fonctions  plus  complexes  de  la  salle  des  operations, 
notamment  les  taches  necessitant  une  interaction  entre  les  equipes  de  guerre  completes,  entre  les 
directeurs  de  conduite  de  la  guerre  et  l’ORO,  ou  entre  l’ORO,  le  cmdt  et  1’ ensemble  de  l’equipe  de  la 
salle  des  operations,  la  SESO  offre  le  meilleur  environnement  pour  les  activites  d’E  et  E.  La  gamme 
de  scenarios  actuels  utilises  pour  la  SESO  et  le  NCOT  peuvent  etre  adaptes  pour  les  E  et  E.  Compte 
tenu  des  demandes  d’ instructions  courantes  de  la  Marine,  la  SESO  (et  peut-etre  egalement  le 
NCOT),  de  meme  que  le  personnel  qui  y  travaille,  sont  rarement  disponibles.  II  est  recommande  de 
n’effectuer  qu’un  essai  afin  de  confirmer  la  pertinence  du  NCOT  pour  les  E  et  E  dans  les  plus  brefs 
delais. 
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Executive  Summary 


This  report  reviews  five  training  simulator  facilities  in  the  Canadian  Navy  for  their  suitability  in 
conducting  test  and  evaluation  (T&E)  in  support  of  the  COMDAT  multi-sensor  data  fusion  (MSDF) 
initiative.  The  facilities  are  the  Operations  Room  Team  Trainer  (ORTT),  the  Naval  Combat  Operator 
Trainer  (NCOT),  the  Combat  Systems  Training  Centre  (CSTC)  and  HALIFAX  Class  Ops  Room 
alongside  or  at  sea.  A  set  of  detailed  capabilities  for  conducting  T&E  was  developed  under  the  four 
major  categories  of:  Logistics,  Experimental  Control,  Data  Collection  and  Data  Storage  and 
Analysis.  Each  of  the  potential  facilities  was  reviewed  against  these  categories  and  the  results 
tabulated.  The  review  took  into  account  earlier  work  developing  a  framework  for  evaluating  military 
information  systems  as  well  as  JDL  data  fusion  levels.  This  framework  considers  the  need  to 
conduct  iterative  evaluations  of  human  performance  during  development  in  terms  of  part,  whole,  and 
team  tasks  at  several  levels  of  prototype  design  under  mock  up,  test  bed  or  simulator,  and  sea  trial 
conditions.  Existing  Navy  systems  for  collecting  performance  data  were  also  reviewed.  The 
capabilities  of  each  facility  were  then  assessed  for  their  suitability  for  conducting  the  expected  range 
of  T&E  activities. 

It  was  concluded  that  the  NCOT  would  be  the  most  suitable  facility  for  conducting  T&E  using 
single  operators  performing  simple  operational  functions  such  as  detecting  and  tracking  radar 
contacts.  While  more  limited  in  its  simulation  fidelity  than  ORTT,  for  repeated  trials  of  simple 
tasks,  the  NCOT  facility  appears  to  be  the  most  versatile  to  configure  and  the  most  economical  to 
operate.  NCOT  allows  the  collection  of  T&E  data  for  several  operators  performing  the  same  task 
in  parallel.  NCOT  also  appears  suitable  for  assessing  performance  of  small  teams  of  2-4 
individuals  performing  more  complex,  interacting  functions.  This  assessment  is  based  upon  the 
expectation  that  the  contractor  responsible  for  NCOT  will  deliver  software  that  supports  this  level 
of  functionality  within  the  next  year. 

For  more  complex  functions  in  the  Ops  Room,  including  tasks  involving  interactions  between 
complete  warfare  teams,  interactions  between  warfare  directors  and  the  ORO,  and  between  the 
ORO,  CO  and  the  entire  Ops  Room  team,  the  ORTT  represents  an  excellent  environment  for 
conducting  T&E  activities.  A  range  of  scenarios  exists  for  the  ORTT  and  for  NCOT,  which  may 
be  adaptable  for  T&E  purposes. 

Severe  constraints  on  the  availability  of  the  ORTT  (and  possibly  the  NCOT)  were  noted,  due  to 
existing  training  demand  from  the  Navy.  Further,  the  necessary  personnel  infrastructure,  as  game 
players  and  controllers,  in  support  the  ORTT  is  high  and  is  also  constrained  by  personnel 
availability. 

It  is  recommended  that  a  simple  proof  of  concept  trial  for  assessing  the  suitability  NCOT  for  T&E 
be  conducted  as  soon  as  possible. 
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Sommaire 


Le  present  rapport  porte  sur  la  pertinence  d’utiliser  les  cinq  simulateurs  d'entrainement  de  la  Marine 
canadienne  pour  effectuer  les  essais  et  revaluation  (E  et  E)  de  l’initiative  de  fusion  de  donnees  des 
multicapteurs  COMDAT  (MSDF).  Les  installations  comprennent  le  simulateur  pour  l'equipe  de  la 
salle  des  operations  (SESO),  le  simulateur  d'operations  de  combat  naval  (NCOT),  le  centre 
^instruction  des  systemes  de  combat  (CSTC)  et  la  salle  des  operations  de  classe  HALIFAX,  a 
proximite  ou  en  mer.  Une  serie  de  capacites  detaillees  permettant  d’ effectuer  les  E  et  E  a  ete 
identifiee  en  fonction  des  quatre  grandes  categories  que  sont  la  logistique,  le  controle  experimental, 
la  collecte  des  donnees  et  l’archivage  et  1’  analyse  des  donnees.  Chaque  installation  a  ete  etudiee  a  la 
lumiere  de  ces  categories  et  leurs  capacites  ont  ensuite  ete  tabul6es.  L’ etude  a  tenu  compte  des 
travaux  precedents  pour  developper  un  cadre  devaluation  des  systemes  d’ information  militaires  de 
meme  que  des  niveaux  de  fusion  de  donnees  JDL.  Le  cadre  tient  compte  de  la  necessite  d’ effectuer 
des  evaluations  iteratives  des  performances  humaines  pendant  le  developpement  en  termes  de 
parties,  d’ ensemble  et  de  taches  d’equipe  a  divers  niveaux  de  prototype  sous  maquette,  banc  d’essai 
ou  simulateur,  et  conditions  d’ essais  en  mer.  Les  systemes  en  service  dans  la  Marine  pour  la  collecte 
de  donnees  de  performance  ont  egalement  ete  etudies.  Les  capacites  de  chaque  installation  ont 
ensuite  ete  evaluees  en  ce  qui  a  trait  a  leur  pertinence  pour  effectuer  toute  la  gamme  prevue  des 
activites  d’E  et  E. 

Le  NCOT  serait  1’  installation  la  plus  appropriee  pour  effectuer  les  E  et  E  tant  avec  des  operateurs 
seuls  qu’avec  des  equipes  restreintes  pour  les  fonctions  operationnelles  simples  telles  que  la 
detection  ou  les  poursuites  de  contacts  radars.  Bien  qu’un  peu  plus  limite  en  ce  qui  a  trait  a  la 
fidelite  de  simulation  que  la  SESO  lors  d’essais  repetes  de  taches  simples,  le  NCOT  est  plus  facile 
a  configurer  et  moins  couteux  a  faire  fonctionner.  II  permet  la  collecte  des  donnees  E  et  E  pour 
divers  operateurs  qui  effectuent  une  meme  tache  en  parallele.  Le  NCOT  convient  egalement  pour 
evaluer  la  performance  de  petites  equipes  de  2  a  4  personnes  effectuant  des  taches  plus  complexes 
avec  des  fonctions  interactives.  L’evaluation  est  conditionnelle  au  fait  que  1’ entrepreneur 
responsable  du  NCOT  doit  foumir  le  logiciel  soutenant  ce  niveau  de  fonctionnalite  en  de<ja  d’un 
an. 

Pour  les  fonctions  plus  complexes  de  la  salle  des  operations,  notamment  les  taches  necessitant  une 
interaction  entre  les  equipes  de  guerre  completes,  entre  les  directeurs  de  conduite  de  la  guerre  et 
l’ORO,  ou  entre  l’ORO,  le  cmdt  et  1’ ensemble  de  l’equipe  de  la  salle  des  operations,  la  SESO  offire 
le  meilleur  environnement  pour  les  activites  d’E  et  E. 

La  gamme  de  scenarios  actuels  utilises  pour  la  SESO  et  le  NCOT  peuvent  etre  adaptes  pour  les  E  et 
E. 

Compte  tenu  des  demandes  d’ instructions  courantes  de  la  Marine,  la  SESO  (et  peut-etre  egalement 
le  NCOT),  sont  rarement  disponibles.  De  plus,  le  personnel  affecte  a  la  SESO  (techniciens  et 
controleurs  notamment)  n’est  pas  toujours  disponible. 

II  est  recommande  de  n’ effectuer  qu’un  essai  afin  de  confirmer  la  pertinence  du  NCOT  pour  les  E 
et  E  dans  les  plus  brefs  delais. 
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1 .  Introduction 


This  report  addresses  the  requirement  in  the  Statement  of  Work  for  DCEEM  Contract  No.  W77 11- 
7-7404-21,  to  review  potential  data  collection  environments  and  existing  data  capture  tools  and 
outputs  for  their  ability  to  support  the  evaluation  of  the  impact  of  future  multi-sensor  data  fusion 
(MSDF)  on  command  decision  making. 

This  report  is  organised  as  follows.  The  first  section  describes  the  general  methodology  and  is 
followed  by  a  section  containing  detailed  descriptions  of  five  simulation  facilities,  the  Operations 
Room  Team  Trainer  (ORTT),  the  Naval  Combat  Operator  Trainer  (NCOT),  the  Combat  Systems 
Training  Centre  (CSTC)  and  HALIFAX  Class  Ops  Room  (alongside  or  at  sea)1.  Systems  developed 
by  the  Navy  to  capture  and  analyse  combat  systems  data  are  also  reviewed  in  this  section.  This 
followed  by  brief  sections  on  availability  of  personnel  for  testing  and  availability  of  existing  Navy 
measures  of  performance  (MOPs).  The  report  concludes  with  a  discussion  and  analysis  of  the 
results  and  recommendations  for  next  steps. 


1  During  the  course  of  gathenng  information  for  this  project  we  learned  that  there  were  other  test  beds  currently  under 
development  at  DREV  for  prototyping  and  evaluating  MSDF  concepts  in  support  of  COMDAT.  The  Scientific  Authonty 
at  DCIEM  has  indicated  that  evaluation  of  these  would  be  out  of  scope  for  present  purposes. 
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2.  Methodology 


2.1  Background 

The  initial  goal  was  to  generate  a  list  of  potential  ideal  requirements  to  perform  the  full  range  of 
Test  and  Evaluation  (T&E)  functions  in  support  of  obtaining  MOPs  to  assess  the  impact  of  MSDF. 
These  requirements  were  derived  from  previous  work  by  Humansystems  Incorporated  (HSI®)  in 
developing  a  generic  framework  for  evaluation  of  C2  systems.  Readers  are  directed  to  the  report 
by  Matthews,  Webb  and  McCann,  1997  for  full  details  of  a  generic  framework  of  evaluation  to 
assess  C2  information  systems,  and  to  more  specific  work  on  the  cognitive  requirements  of  the 
Operations  Room  Officer  (ORO)  of  a  Canadian  Patrol  Frigate  (CPF)  (Matthews,  Webb  and  Bryant, 
1999).  In  addition,  the  limited  available  literature  concerning  C2  MOPs  was  consulted. 

The  general  evaluation  model  is  shown  below. 


JOB  EVALUATION  COGNITIVE  EVALUATION  EVALUATION  EVALUATION  EVALUATION 

POSITIONS  FOCUS  ANALYSIS  DOMAINS  MEASURES  SCENARIOS  CONFIGURATIONS 

Figure  1:  Conceptual  Model  for  Test  and  Evaluation  (from  Matthews  et  al  (1997) 


2.2  Evaluation  Domain 

Using  the  above  as  an  overall  guide,  it  was  believed  that  the  overall  focus  of  work  for  evaluating 
MSDF  for  COMDAT  will  primarily  be  in  the  user  fit  and  operational  impact  domains.  This  is 
based  on  the  expectation  that  the  commercial  developer  of  MSDF  software  will  have  conducted  the 
T&E  steps  to  ensure  appropriate  levels  of  function  usability  early  in  system  and  concept 
development,  through  assessment  methodologies  such  as  storyboarding  and  cognitive 
walkthroughs.  Since  the  goal  of  the  present  work  is  on  developing  MOPs  that  are  tied  to  the  actual 
performance  of  operational  tasks,  the  primary  focus  of  the  present  analysis  will  be  on  configuration 
stages  two  and  three  as  outlined  below  in  Figure  2. 

It  should  be  noted  that  the  test  beds  under  development  at  DREV  may  be  a  suitable  environment 
for  conducting  early  assessments  and  proof  of  concept  for  MSDF  prototype  displays  and 
functionality.  These  test  beds  may  also  evolve  into  systems  that  will  accommodate  some  of  the 
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requirements  for  stage  two  assessment.  Cross  validation  of  data  derived  from  these  different 
facilities  should  be  a  priority. 


2.3  Evaluation  Configuration 


Three  major  types  of  configuration  have  been  identified,  as  illustrated  below. 


Stage  1:  Story  Board 

HF/Domain  SMEs 
Task  walkthrough 
Focus  groups 
Rating  scales 
Check  lists 

Stage  2:  Test  Bed 

Part  context 
Part  system 
Speed  and  accuracy 
Individual  &  team 


Team 


Stage  3:  Full  Scale 
Simulation  or  Sea  Trial 

Whole  context 
Whole  system 
Parallel  tasking 


Figure  2:  Three  Evaluation  Configurations 

It  is  expected  that  the  present  T&E  objectives  will  require  a  range  of  configurations  from  a  simple 
test  bed,  through  more  complex  test  beds  up  to  a  full  scale  simulation  or  sea  trial.  The  full  scale 
simulation,  as  described  below,  would  appear  to  be  the  primary  candidate  for  evaluating  the  impact 
of  MSDF  at  the  levels  of  the  warfare  directors  and  above. 

* The  primary  purpose  of  this  configuration  is  to  determine  if  the  system  will  deliver 
acceptable  operational  performance  under  operational  conditions.  This  will  involve 
the  achievement  of  more  complex  system  functions  and  sub-goals  at  the  whole 
system  level,  with  fully  trained  users,  and  with  the  system  in  its  final  configuration  and 
layout.  This  also  implies  the  presence  of  concurrent  and  multiple  tasks  at  the 
individual,  team  and  crew  level.  Evaluation  within  this  configuration  should  include  not 
only  the  hardware  and  software  components  of  the  system  but  also  training  packages, 
layout  and  organisational  structure."  Matthews  et  al  (1997). 

Although  the  original  report  placed  the  full-scale  simulator  and  field  trial  environments  in  the  same 
category,  for  present  purposes  it  may  be  better  to  distinguish  between  the  particular  T&E 
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characteristics  of  each.  As  shall  be  shown  later,  the  ORTT,  CSTC  and  Ops  Rooms  alongside  might 
all  be  considered  to  be  examples  of  a  high  fidelity,  full  scale  simulation,  the  NCOT  represents  a 
highly  advanced  level  test  bed  and  the  Ops  Room  ship-at-sea  represents  the  unique  environment  of 
a  sea  trial. 


2.4  Fidelity 

In  considering  the  requirements  for  fidelity,  we  were  guided  by  the  following  recommendations  in 
Matthews  et  al  (1997)  concerning  the  various  test  bed  environments. 

"Potential  levels  of  fidelity  and  completeness  within  this  stage  range  from  rudimentary 
to  a  full-scale  prototype.  At  one  end  of  the  range,  there  are  tasks  and  sub-tasks  that 
require  the  interaction  between  one  operator  and  a  single  console.  At  the  simplest 
level,  this  could  simply  be  a  single  screen,  depicting  a  system  state  to  which  the  user 
has  to  respond  with  limited  options.  At  the  other  end  of  the  range,  the  system 
software  has  the  capability  of  generating  dynamic  events,  sequences  of  events  and  a 
range  of  operator  input  functions.  In  terms  of  physical  configuration,  elements  such  as 
workstation,  console  and  floor  layout  can  be  considered  in  the  context  of  a  command 
post  or  operations  centre,  vehicle,  team  or  other  operational  configuration. 

Relevant  levels  of  fidelity  would  include  not  only  the  appearance  and  dynamics  of  the 
software  and  hardware,  but  also  the  degree  of  realism  with  respect  to  the  overall 
operational  context  in  terms  of  volume  and  complexity  of  information,  shift  duration, 
lighting  and  other  environmental  conditions,  and  team  operations. " 

Thus,  there  is  no  single  requirement  for  fidelity.  Instead,  it  will  vary  depending  upon  the 
complexity  of  the  operational  function  under  scrutiny. 


2.5  Scenarios 

A  final  point  of  guidance,  drawn  from  Matthews  et  al  (1997)  was  the  need  to  identify  scenario 
elements  that  should  be  consistently  applied  across  the  range  of  T&E  environments,  recognising 
that  the  degree  of  sophistication  in  implementing  these  increases  across  the  three  evaluation  stages. 
While  this  consistency  was  initially  discussed  in  the  context  of  the  progressive  evaluation  of  new 
systems,  the  general  principle  of  having  standard,  validated  tasks  that  can  be  accurately  specified 
and  benchmarked  using  statistically  appropriate  sample  sizes,  applies  equally  to  the  need  of 
evaluating  the  potential  impacts  of  MSDF  against  existing  human-systems  performance. 

2.6  Criteria  for  evaluating  the  test  environments 

2.6.1  T&E  Measurement  approaches 

Based  upon  above,  it  is  anticipated  that  there  will  need  to  be  at  least  five  different  T&E  paradigms, 
as  follows: 

•  Individuals  doing  part  tasks,  tasks  and  task  strings  in  isolation  (possibly  using  multiple 
subjects  to  collect  data  in  parallel) 

•  A  small  team  of  subjects  (e.g.  a  warfare  area)  performing  mission-like  tasks  with 
interaction  between  team  members 

•  Team  interacting  with  each  other 
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•  ORO  and  warfare  directors  performing  mission  tasks. 

•  The  entire  Ops  Room  team  performing  small  and  large  scale  mission  scenarios 
Hence  the  particular  test  environments  will  need  to  accommodate  this  range  of  approaches. 

2.6.2  Anticipated  T&E  Measures 

Evaluation  measures  are  expected  to  include:  performance  speed  and  accuracy  in  completing 
scenario-based  sub-tasks,  tasks  and  task  strings;  error  patterns;  timeliness  of  actions;  measures  of 
situation  awareness  using  probe  and  other  techniques;  decision  quality  and  speed;  measures  of 
communication  effectiveness,  communication  load  and  estimates  of  mental  workload. 

To  achieve  this  will  require  capabilities  for: 

•  Video  and  audio  recording  of  users  actions  and  communications 

•  Logging  of  ongoing  system  events, 

•  Capture  of  user  responses  to  injected  probe  messages, 

•  Capture  of  keyboard,  switch  or  track  ball  actions  that  signify  the  start  and  end  of  tasks. 

Specific  measures  will  include  time  and  accuracy  to  complete  mission  goals  and  sub-goals, 
successful  completion  of  critical  tasks,  individual  and  team  ratings  of  workload  and  situation 
awareness,  and  measures  of  team  co-ordination  and  effectiveness.  At  the  levels  of  teams,  evaluation 
will  involve  more  complex  scenarios  that  involve  the  successive  completion  of  multiple  tasks  by 
several  users  interacting  as  a  team.  Measures  will  include  communication  effectiveness  and  patterns, 
assessment  of  team  situation  awareness  and  evaluation  of  team  co-ordination. 

For  more  details  of  the  general  requirements  for  analysing  the  required  logistical  support  for  T&E 
MOP  data  collection,  the  reader  is  referred  to  Annex  A  of  Matthews  et  al  (1997)  which  describes 
the  range  of  measures  required.  For  present  purposes,  MOPs  relevant  to  test  beds  and  field  trials 
have  been  extracted  from  this  and  are  shown  in  Annex  A. 

2.6.3  Analysis  of  requirements 

Taking  into  account  the  measurement  approaches  and  anticipated  measures,  four  major  categories 
of  requirements  were  identified  as  follows.  Each  of  these  was  then  broken  down  into  more  detailed 
requirements,  as  outlined  later  in  this  section. 

1.  Logistics 

2.  Experimental  Set-up  and  Control 

3.  Data  Collection 

4.  Data  Storage  and  Analysis 

These  detailed  requirements  were  then  used  to  provide  a  common  basis  for  describing  and 
evaluating  each  potential  test  environment. 

There  was  considerable  discussion  concerning  the  development  and  use  of  criteria  for  evaluating  the 
different  environments  and  whether  a  formal  quantitative  approach  for  comparing  candidate  T&E 
environments  should  be  adopted.  Such  an  approach  might  involve  generating  a  score  for  each  item 
in  the  requirements  list  and  then  aggregating  the  scores  using  a  weighted  sum.  In  practice,  this  would 
mean  that  at  the  level  of  the  individual  requirement  it  would  be  necessary  to  establish  precise  criteria 
for  each  entry  in  the  table  (e.g.  "noise  level  in  test  facility  is  plus  or  minus  10  db  from  actual  ship 
Ops  Room").  For  other  requirements  it  would  be  necessary  to  generate  a  rating  scale  for  the  degree 
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to  which  they  met  the  T&E  needs.  Then,  in  order  to  obtain  an  overall  estimate  of  capability  of  each 
environment,  a  weighting  scheme  might  be  devised  to  combine  the  individual  requirement  items  to 
generate  an  single  summary  evaluation  score  that  would  allow  at  least  a  rank  ordering  of  the 
alternatives. 

In  the  end  this  approach  was  not  pursued  for  a  number  of  reasons.  First,  the  question  of  interest  is 
not  which  facility  is  better  overall  for  meeting  all  of  the  T&E  requirements,  since  there  is  no  single 
T&E  paradigm.  For  example,  the  fundamental  needs  to  support  T&E  activities  for  individuals 
doing  part  tasks,  tasks  and  task  strings  in  isolation  are  clearly  quite  different  from  what  is  required 
for  ORO  and  warfare  directors  performing  mission  tasks,  or  the  entire  Ops  Room  team  performing 
small  and  large  scale  mission  scenarios.  Thus,  it  would  be  impossible  to  develop  a  weighting 
scheme  that  would  have  appropriate  validity  for  all  of  these  situations. 

A  second  reason  for  not  developing  a  quantitative  evaluation  metric  for  each  T&E  capability  is  that 
it  would  have  involved  a  level  of  precision  that  was  unnecessary.  For  most  of  the  items  in  the 
requirements  list,  it  is  not  necessary  to  know  if  the  facility  meets  the  need  optimally  but  rather 
whether  it  satisfies  minimally,  i.e.  suffices. 

A  third  reason  is  that  the  level  of  effort  required  in  developing  a  quantitative  evaluation  metric, 
given  that  it  would  be  unproven  and  unvalidated,  being  simply  based  upon  the  opinions  of  the  HSI® 
team.  To  create  the  necessary  validity  would  require  a  level  of  effort  that  would  inappropriately 
consume  budgetary  resources  that  are  assigned  elsewhere  in  the  contract. 

Therefore  the  approach  chosen  was  to  assess  each  environment  in  terms  of  what  specific 
approaches  and  methodologies  to  T&E  (i.e.  areas  indicated  in  section  2.6.1)  would  best  be 
supported  by  the  facility.  In  order  to  determine  whether  a  particular  test  environment  contained  the 
range  of  required  capabilities  to  conduct  T&E,  it  was  assessed  against  the  following  list  of  core 
requirements  that  would  need  to  be  fulfilled  if  the  system  were  to  be  considered  as  a  viable  T&E 
environment. 

These  requirements  were  developed  as  a  result  of  discussion  among  the  HSI  team  members  and 
based  upon  previous  experience  in  developing  and  executing  T&E  activities  within  military 
contexts.  Each  of  these  items  was  considered  to  be  essential  for  the  successful  conduct  of  T&E, 
with  the  possible  exception  of  the  item  "  Capability  for  T&E  personnel  to  create  scenarios 
autonomously  (with  training)".  This  could  be  considered  to  be  a  highly  desirable  requirement 
rather  than  essential,  in  that  it  has  the  potential  to  reduce  the  overhead  in  time  and  labour  for 
scenario  production.  In  essence,  if  the  facility  were  not  able  to  meet  a  single  one  of  the  following 
criteria  (with  the  exception  of  the  above),  it  would  be  considered  to  have  serious  limitations  or  be 
unusable  as  an  appropriate  T&E  environment. 

The  core  requirements  are: 

Logistics: 

•  Availability  of  the  facility 

•  Appropriate  Workstation  Functionality  (to  the  specific  T&E  paradigm) 

•  Flexible  arrangement  for  workstation/controller  combinations 

•  Capability  to  simulate  the  functions  required  for  the  team  role  or  roles  under  analysis 
Experimental  Set-up  and  Control 

•  Capability  to  create  T&E  scenarios 
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•  Capability  for  T&E  personnel  to  create  scenarios  autonomously  (with  training) 

•  Capability  to  control  T&E  scenarios  in  real  time  (e.g.  insert  probes,  messages) 

Data  Collection 

•  Capability  to  capture  data  to  the  required  level  of  precision 

•  Capability  to  capture  data  reliably 
Data  storage  and  analysis 

•  Capability  to  capture  the  required  volume  of  data 

•  Capability  for  reducing  and  analysing  the  data 

For  each  of  these  items,  more  specific  and  detailed  questions  were  developed  in  order  to  derive  a 
better  understanding  of  the  capability  of  each  environment.  For  example,  under  the  general 
heading  of  " Availability ",  it  was  important  to  understand  issues  such  as  the  booking  process,  the 
required  lead  time  for  booking,  the  ongoing  usage  pattern,  whether  the  facility  could  be  used 
exclusively  for  T&E  and  would  need  to  be  shared,  and  the  general  level  of  availability,  e.g  24/7. 

The  result  of  this  process  was  a  comprehensive  listing  of  questions  and  issues  relating  to  the 
capabilities  and  these  are  shown  in  full  in  Annex  B.  This  listing  provided  the  basis  for  a  common 
set  of  questions  to  be  asked  on  the  site  visits  when  evaluating  each  of  the  facilities  of  interest. 

2.6.4  Site  Visits 

Following  development  of  the  requirements  and  associated  question  items,  a  site  visit  was 
undertaken  from  February  18-22,  2001  to  view  each  of  the  potential  test  environments  (with  the 
exception  of  the  Ops  Room  at  Sea).  Four  members  of  the  HSI  team  took  part  in  the  visit  and 
approximately  2-4  hours  was  spent  at  each  site.  This  was  supplemented  with  meetings  with 
relevant  SMEs  to  address  the  topic  areas  of  the  questionnaire  in  Annex  B. 

Wherever  possible,  documentation  appropriate  to  the  facility  was  obtained  or  requested  for  later 
delivery. 

As  part  of  the  overall  visit,  HSI  was  provided  with  a  lengthy  briefing  on  existing  data  capture  and 
analysis  methods  and  an  overview  of  sample  exercise  data. 
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3.  Description  of  Facilities 


As  a  result  of  the  site  visits,  the  review  of  available  documentation,  discussions  and  follow-up 
questions  with  local  SMEs,  answers  to  the  detailed  questions  listed  in  Annex  B  were  obtained. 
There  were  a  few  gaps  in  the  answers  where  it  has  been  impossible  to  obtain  the  required 
information.  These  answers  and  any  relevant  comments  are  shown  in  detail  in  Annex  C  for  each  of 
the  facilities. 

What  follows  in  this  section  is  a  detailed  description  of  each  of  the  potential  test  environments 
based  upon  a  distillation  of  the  answers  to  the  detailed  questions.  For  convenience,  each  facility  is 
described  separately  in  a  different  sub-section,  under  the  following  common  headings: 

■  Overall  Description 

■  Simulation  capability 

■  Scenario  Development 

■  Real-time  control  and  monitoring  of  scenarios 

■  Data  recording 

■  Data  replay  and  analysis 

■  Availability 

■  Support  for  scenario  and  test  development 

■  Support  for  running  T&E  scenarios 

■  Support  for  data  analysis 

■  Summary  of  advantages  and  disadvantages 

It  is  suggested  that  having  read,  the  summary  descriptions  of  capabilities  for  each  facility,  the 
reader  consuls  Annex  C  for  detailed  information  concerning  any  specific  issue. 

It  should  be  noted  that  the  issue  of  availability  of  personnel  to  act  as  T&E  subjects  is  not  discussed 
within  the  context  of  each  facility  separately,  since  the  issues  are  essentially  the  same  for  all.  A 
separate  section  deals  with  this. 

3.1.  ORTT 

The  descriptions  below  are  largely  quoted  directly  from  descriptive  documentation  provided  on  the 
ORTT  facility  (ORTT  SOM  on  CD),  supplemented  by  information  and  photographs  obtained 
during  the  site  visit. 

3.1 .1 .  Overall  description 

The  ORTT  training  complex  consists  of  a  large  simulation/training  system  used  for  team  training 
of  HALIFAX  Class  operations  personnel.  The  ORTT  is  an  interactive,  computer-based  trainer 
capable  of  simulating  two  ownships,  multiple  other  vehicles  (air,  surface,  and  sub-surface),  and  the 
weapons,  sensors,  and  environmental  effects  needed  to  support  Ops  team  training.  The  trainer 
(shown  in  Figure  3)  consists  of  two  functional  HALIFAX  Class  Ops  Room  mock-ups  (SimOps, 
labelled  as  CPF  Cubicle  in  Figure  3),  associated  Bridge  mock-ups,  a  Training  Control  and  data 
capture  facility,  and  a  Brief/Debrief  facility. 
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The  simulated  bridge  is  used  primarily  as  a  source  to  input  data  and  receive  communications  from 
the  simulated  Ops  Room.  The  bridge  uses  a  large  three-screen  visual  display  to  project  the 
simulated  ocean  and  air  environments,  and  ship  profiles  as  would  be  seen  from  a  typical  bridge. 
The  display  area  is  somewhat  narrower  in  coverage  than  in  reality.  Various  degrees  of  sea  state 
and  environmental  conditions  (e.g.  fog)  can  be  simulated.  The  displays  provide  a  level  of  realism 
that  is  probably  adequate  for  providing  data  for  input  to  scenarios,  if  required.  A  workstation 
simulates  the  basic  ship  control  capabilities  and  provides  a  communication  link  by  headset/voice 
and  data  to  the  Ops  Room. 

Each  simulated  Ops  room  (SimOps)  allows  the  Ops  team  members  to  receive  training  through  the 
performance  of  their  normal  duties  using  equipment  that  accurately  simulates  the  HALIFAX  Class 
Ops  Room  equipment  suite.  Each  room  provides  a  training  environment  through  the  provision  of 
operator  consoles,  control  panels,  and  indicators,  all  located  in  accordance  with  the  basic  shipboard 
HALIFAX  Class  Ops  Room  and  bridge  layouts.  The  cubicle  consoles,  panels  and  indicators  also 
possess  the  physical  and  functional  fidelity  associated  with  the  HALIFAX  Class  Ops  Room 
operator  machine  interfaces  (OMIs).  The  two  SimOps  can  be  interactive,  with  both  representing 
individual  "ownships"  within  the  same  training  exercise.  The  SimOps  also  contains  versions  of  the 
usual  stateboards,  although  the  exact  characteristics  and  locations  are  not  identical  to  that  of  a 
typical  HALIFAX  Class  Ops  Room.  Examples  of  a  single  ORTT  workstation  and  a  group  of 
workstations  are  shown  below.  Note  the  high  level  of  physical  fidelity  of  the  workstation  itself. 
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Figure  4:  A  single  ORTT  Workstation 


Figure  5:  ORTT  Back  Row  -  from  position  of  CO 


Each  SimOps  contains  the  following  simulated  but  functional  equipment  suites: 

a.  Combat  Operator  Positions  -  Operator  stations  are  provided  for  the  Commanding 
Officer  (CO),  Operations  Room  Officer  (ORO),  Sensor  Weapons  Controller  (SWC), 
Assistant  Sensor  Weapons  Controller  (ASWC),  Electronic  Warfare  Supervisor  (EWS), 
Shipboard  Air  Controller  (SAC),  two  Radar  Trackers  (RT 1  and  RT2),  Track 
Supervisor  (TS),  Sonar  Control  Supervisor  (SCS),  and  two  Naval  Communicators 
(NAV  COMMS).  Each  of  these  stations  provides  an  operating  environment  within 
which  the  respective  operators  can  perform  their  job  tasks.  As  is  the  case  in  an  actual 
HALIFAX  Class  Ops  Room,  the  Operations  Room  Supervisor  (ORS)  has  no  dedicated 
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station  in  the  SimOps,  but  the  operating  environment  realistically  represents  the  ORS's 
shipboard  environment. 

b.  Officer  of  the  Watch  (OOW)  Position  -  The  OOW  position  provides  the  environment 
required  for  OOW  operations. 

c.  Ship's  Ops  Room  Indicators  -  Information  normally  provided  by  ship's  bulkhead 
indicators  is  displayed  in  each  ORTT  Ops  Room. 

d.  Communications  -  Internal  and  external  communications  for  each  team  member  within 
each  cubicle  facility  are  provided  through  a  physically  and  functionally  simulated 
Shipboard  Integrated  Communications  System  (SHINCOM),  with  personnel  acting  as 
role  players  to  generate  the  required  input  messages  to  scenarios. 

e.  Command  and  Control  System  (CCS1)  -  The  ORTT  supports  the  loading  and  execution 
of  ship's  CCS  software.  Execution  of  ship's  CCS  software  provides  for  the  highest 
possible  level  of  system  fidelity  for  all  operators. 

The  SimOps  facility  does  not  contain  workstations  for  CANEWS,  CANT  ASS  or  GCCS,  but 
simulated  data  from  these  sources  are  provided. 

The  Training  Control  Facility  (TCF)  provides  the  external  interfaces  to  the  SimOps  cubicles  for 
simulation  and  control  and  is  physically  separate  from  the  SimOps.  The  TCF  provides  the 
instructor  staff  with  the  capability  to  set  up  training  scenarios,  control  and  monitor  the  progress  of 
the  training  sessions,  and  subsequently  debrief  the  trainee  team.  For  the  real  time  control  of  the 
training  sessions,  trained  personnel  are  required  to  act  as  game  players  to  move  gamepieces  and  to 
act  as  communicators  who  provide  appropriate  input  messages  and  respond  to  messages  from  the 
team  in  the  SimOps.  Training  Control  consists  of  the  hardware  and  software  and  OMIs  necessary 
to  create  a  realistic  Ops  Room  training  environment.  Training  Control  provides  sensor,  weapon  and 
navigation  subsystem  simulation,  and  realistically  models  the  impact  that  ownship  motion,  and  the 
external  ocean,  meteorological  and  tactical  environments  have  on  ships'  sensors..  Instructor  stations 
are  provided  for  the  instructor  staff  to  monitor  and  control  the  training  exercises.  Each  Ops  Room 
WS  display  can  be  monitored  separately  on  a  slave  WS  in  the  TCF.  Training  staff  stations  are 
provided  for  operators  to  participate  in  the  exercise  under  the  direction  of  the  instructors  or  the  staff 
manning  the  mock-ups. 

These  above  capabilities  are  supported  by  the  following: 

•  Instructor  Stations  are  provided  for  the  Tactics  and  Team  Training  Officer  (TTTO), 
Anti-Surface  Warfare  Tactical  Instructor  (TAC  ASuW),  Anti- Air  Warfare  Tactical 
Instructor  (TAC  AAW),  Anti-Submarine  Warfare  Tactical  Instructor  (TAC  ASW), 
Tactical  Air  Control  Instructor  (TAC  AIR),  and  the  Command  and  Control  Information 
System  Tactical  Instructor  (TAC  CCIS). 

•  Training  Staff  Stations  are  provided  for  personnel  designated  to  assist  the  instructor 
complement.  These  stations  include  the  Gamepiece  Operators  (GOs),  Trainer  Control 
Officer  (TCO),  Trainer  Control  Supervisor  (TCS),  Audio  Visual  Operator  (AVO),  and 
a  Trainer  Support  Operator  (TSO). 

•  Support  Stations  are  provided  for  the  CIO,  Electronic  Support  Measures  (ESM), 
Separate  Tracking  and  Illuminating  Radars  (STIRs),  Hull-Mounted  Sonar  (HMS), 
Sonobuoy  Processing  System  (SPS),  Canadian  Towed  Array  Sonar  System 
(CANT ASS)  and  other  ship  outstation  functions.  Each  of  these  support  stations 
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provides  the  necessary  system  and  voice  stimuli  to  each  cubicle's  Combat  System.  The 
Support  Stations  include  the  Ownship  Support  Specialist  (OSS),  Fire  Control 
Specialist  (FCS),  Acoustic  Warfare  Support  Specialist  (AWS),  EW  Support  Specialist 
(EWS),  and  a  Signals  Operator  (SigOp). 

3.1 .2.  Simulation  capability 

The  ORTT  provides  the  capability  for  officers  and  Non-Commissioned  Members  (NCMs)  of  the 
HALIFAX  Class  Ops  Room  to  learn,  practise  and  enhance  those  skills  that  create  an  effective 
Combat  Team  in  a  simulated,  multi-threat,  multi-platform,  time-stressed  environment.  The  ORTT 
presents  real-time  dynamic  simulations  of  tactical  engagements  to  exercise  the  Combat  Team  in 
procedures,  operations  and  the  execution  of  tactics  in  a  variety  of  situations  that  may  occur  during 
peace  or  wartime  conditions. 

The  ORTT  provides  a  realistic  training  environment  by  simulating  Ops  Room  external  interfaces 
wherein  individual  team  members  are  permitted  to  perform  their  normal  duties  using  equipment 
that  reflect  the  functionality  of  the  Halifax  Class  Ops  Room  suite  of  equipment.  The  ORTT 
provides  the  team  with  actual  ship  Combat  System  software  and  operator  interfaces  to  create  a  high 
fidelity  operating  environment.  Inputs  to  the  Combat  System  support  exercising  the  combat  sub¬ 
teams  (Anti-Air  Warfare  (AAW),  Anti-Surface  Warfare  (ASuW),  Anti-Submarine  Warfare 
(ASW))  and  the  overall  Command  Team  in  the  detect  to  engage  sequence  in  multi-threat 
environments.  Simulated  natural  and  tactical  environments  are  provided  to  support  each  HALIFAX 
Class  sensor  and  weapon  system  in  a  synchronised  and  co-ordinated  fashion. 

The  ship  operating  environment  is  also  enhanced  by  providing  internal,  external,  and  data  link 
communications.  Capabilities  to  support  voice  communications  between  the  two  ownships  and 
other  ships,  aircraft,  and  submarines  along  with  voice  communication  with  ownship  stations 
external  to  the  Ops  Room  (flight  deck,  AN/SLQ-25  Torpedo  Countermeasures  Transmitting  Set 
(NIXIE)  winch,  etc.)  are  provided. 

The  ORTT  provides  a  natural  environment  in  which  either  or  both  ownships  operate.  Training 
Control  provides  the  capability  to  define  the  natural  environment  by  setting  the  game  time  and 
specifying  the  gaming  area  and  land  masses,  the  meteorological  and  ocean  environments,  and  the 
sea  conditions  that  exist  within  the  gaming  area.  Unlike  the  CSTC  (as  we  shall  see  later)  the  ORTT 
provides  the  capability  to  degrade  (under  T&E  control)  sensor  and  communications  data  to  reflect 
the  kind  of  loss  or  ambiguity  that  might  be  expected  in  an  actual  operational  context. 

The  Natural  Environment  includes  the  following  features: 

•  Meteorological  Environment  -  Used  to  establish  the  atmospheric  conditions  detectable 
by  ownship  and  which  affect  ownship  sensors. 

•  Geographical  Environment  -  Used  to  define  the  gaming  area,  complete  with  landmass 
inputs  and  to  orient  ownship  according  to  pre-designated  geographic  position. 

•  Oceanic  Environment  -  Used  to  define  the  oceanic  conditions  that  are  detectable  by 
ownship. 

•  Temporal  Environment  -  Used  to  establish  the  timeline  for  the  game. 

The  tactical  environment  provided  by  the  ORTT  includes  air,  surface  and  subsurface  contact 
environment,  the  EW  environment,  the  acoustic  environment,  and  the  communications 
environment.  Tactical  environment  definition  enables  the  user  to  define  the  air,  surface,  and 
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subsurface  gamepieces  with  which  ownship  interacts.  Gamepieces  refer  to  all  vehicles  (air,  surface, 
or  sub-surface),  deployed  weapons  (missiles,  torpedoes,  shells,  mines,  rockets),  deployed  sensors 
(sonobuoys.  Expendable  Bathythermograph  (XBT)  and  Expendable  Sound  Velocimeter  (XSV) 
probes,  towed  array  sonar  systems),  and  deployed  countermeasures  (chaff  clouds  and  flares)  in  the 
gaming  area.  Gamepieces  are  fully  independent  and  operational  entities  within  the  gaming  area  and 
possess  physical  characteristics  to  support  their  detection  and  identification  by  ownship  sensors. 

The  EW  environment  refers  to  all  non-communications  electromagnetic  spectrum  emissions  in  the 
gaming  area.  Each  ownship  views  the  electronic  emissions  of  the  other  ownship  under  the  same 
conditions  as  the  emissions  of  any  other  target.  The  elements  that  make  up  the  EW  environment  are 
the  vehicle  borne  electronic  frequency  sensors  and  weapons  in  addition  to  the  target  radar  cross- 
sections  and  target  radar  aspect  ratios  associated  with  each  type  of  target. 

The  acoustic  environment  refers  to  all  vehicular,  sensor,  and  weapon  acoustic  emissions  in  the 
gaming  area.  Each  ownship  will  perceive  the  acoustics  emissions  of  the  other  ownship  when 
appropriate.  The  elements  that  make  up  the  acoustic  environment  are  the  vehicle  borne  acoustic 
sensors  and  countermeasures  in  the  gaming  area  in  addition  to  the  acoustic  signatures  associated 
with  appropriate  gamepieces. 

The  communications  environment  refers  to  all  communications  (electromagnetic)  frequency 
activity  in  the  gaming  area.  The  elements  that  make  up  the  communications  environment  are  the 
vehicle  borne  communications  systems  including  Low  Frequency/Medium  Frequency/High 
Frequency/Very  High  Frequency/Ultra  High  Frequency  (LF/MF/HFA^HF/UHF)  and  Satellite 
Communications  (SATCOM)  systems.  It  also  includes  the  tactical  data-exchange  radio  link 
environment  to  support  ownship  Link-1 1  communications  between  reporting  and  participating 
units  to  produce  a  consolidated  tactical  Link-1 1  picture. 

3.1.3.  Scenario  Development 

The  ORTT  can  be  used  in  Preparation  Mode  to  prepare  a  game  prior  to  conducting  training.  In  the 
preparation  mode  the  scenario  developer: 

•  Identifies  gamepieces  to  be  used  in  a  scenario 

•  Uses  the  Gamepiece  Editor  to  assign  the  required  communication,  sensor  and 
countermeasure  equipment  to  the  gamepieces 

•  Uses  the  Gamepiece  Editor  to  define  the  manoeuvring  characteristics  of  the 
gamepieces 

•  Uses  the  Profile  Editor  to  define  natural  environment  characteristics 

•  Uses  the  Marking  Editor  to  define  call-signs  and  Link-1 1  Participating  Units  (PUs) 

•  Uses  the  Flight  Plan  Editor  to  define  flight  plans  used  for  missile  trajectories 

•  Uses  the  SAF  Editor  to  place  gamepieces  to  their  initial  positions  at  game  start 

•  Uses  the  Game  Editor  to  define  ownship  characteristics 

•  Specifies  the  required  database  when  saving  the  game  for  use  in  future  training 
sessions. 

In  general  scenario  definitions  are  developed  using  the  tools  provided  above  using  a  custom  OMI. 

The  ORTT  can  also  be  used  in  Validation  Mode  to  validate  a  game  previously  developed.  The 
Validation  mode  allows  the  user  to  run  all  the  elements  of  a  game,  excluding  ownship,  to  ensure 
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that  events  occur  as  desired.  The  game  may  be  started  at  any  specified  time  in  the  scenario  and  may 
be  run  at  any  selected  speed. 

A  number  of  sample  scenarios  were  provided  (e.g.  "Clockwork  Mouse")  and  these  appear  to  have 
some  of  the  necessary  elements  for  the  development  of  specific  T&E  requirements  for  sub-tasks, 
tasks  and  task  strings. 

3.1.4.  Real-time  control  and  monitoring  of  scenarios 

The  ORTT  provides  six  tactical  stations,  used  by  tactical  training  instructors,  to  evaluate  the 
trainees  during  simulated  exercises.  The  user  may  select  any  one  of  the  operations  room  console 
and  monitor  the  trainee's  actions  in  real-time.  These  stations  allow  the  instructors  to  perform  the 
following  while  monitoring  the  trainees: 

•  Determine  whether  mission  success  has  been  achieved 

•  Evaluate  threat  knowledge 

•  Assess  the  warfare  engagement  procedures 

•  Assess  the  trainees  in  the  use  of  air  control  procedures  and  performances 

•  Compile  the  tactical  picture 

•  Disseminate  information  relayed  from  the  training  cubicles 

•  Assess  the  execution  of  tactical  and  operational  procedures 

•  Assess  the  employment  of  weapons,  sensors,  counter-measures  and  Command  and 
Control  System  (CCS)  uses 

Real-time  override  capabilities  exist  for  many  functions  to  change  parameters  from  those  provided 
by  the  scenario  database. 

The  controller  may  display  a  number  of  parameters  from  the  SimOps  workstation  which  is  being 
controlled  including: 

•  CCS  Data 

•  CCS  Tactical  Picture 

•  Link- 11 

•  Synthetic  Environment 

•  Trainee  Station 

•  Panels 

The  CCS  Tactical  Picture  option  provides  for  real-time  monitoring  and  recording  of  the  ownship 
tactical  situation  as  held  by  the  CCS  Global  Data  Base  (GDB).  It  allows  the  selective  display  of  the 
tactical  picture,  as  compiled  by  the  CCS,  with  various  filters,  range  scales  and  offsets.  The  tactical 
picture  is  presented  inside  a  Tactical  Situation  Area  (TSA)  using  standard  Naval  Tactical  Data 
System  (NTDS)  symbols  to  display  tracks.  Note,  however,  that  the  CCS  Tactical  Picture 
application  is  passive  in  nature,  so  it  is  not  possible  to  inject  data  in  the  system  (such  as  creating, 
deleting  or  modifying  tracks,  or  changing  parameters). 

The  Link-11  Tactical  Picture  display  allows  training  staff  to  display  the  Link-11  Synthetic 
Environment  picture  (air,  surface  or  sub-surface).  This  capability  allows  the  selection  of  various 
filters,  range  scales,  offsets  and  graphic  overlays  that  are  used  to  optimise  viewing  of  a  selected 
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picture.  This  display  capability  is  passive  in  that  only  the  representative  display  of  the  source  data 
is  managed  and  manipulated. 

Real-time  monitoring  and  recording  of  events  that  occur  on  the  CCS  Serial  Data  Bus  (SDB) 
network  is  also  available.  It  has  the  same  role  as  the  History  Recording  (HR)  module  in  CCS, 
except  that  all  the  events  are  displayed  in  tabular  format.  The  following  events  are  monitored  and 
recorded: 

•  Quick  Reaction  (QR)  data 

•  Threat  rank  data 

•  Weapons  orders 

•  Operator  entries  and  Quick-Action  Button  (QAB)  selections 

•  CCS  interface  data 

•  Subsystem  status 

•  System  and  console  alarms 

•  Zippo  status 

•  Tactical  picture 

For  each  cubicle  of  the  SimOps  the  following  type  of  events  and  environment  can  be  monitored: 

•  Mission  direction  events 

•  Track  management  events 

•  Threat  parameter  events 

•  Electronic  warfare  events 

•  Weapons  related  events 

•  Surface  operations  events. 

•  Console  Events  -  the  operator's  entry  performed  on  a  SimOps  console. 

•  Status  Events  -  monitoring  of  system  status  events  of  the  selected  console. 

Faults  may  also  be  induced  on  all  sensor,  weapons  and  control  systems. 

Controllers  also  have  a  capability  to  manage  and  modify  tracks  in  the  Link-1 1  simulation.  The 
controller  can  also  modify  bearing,  special  point  and  Area  of  Probability  (AOP)  information  by 
hooking  the  tracks  and  reselecting  the  QABs  used  to  create  them.  The  controller  can  also  control 
and  modify  air  tracks 

The  full  range  of  internal  and  external  voice  communications  can  also  be  simulated  and  there  is  a 
control  feature  to  simulate  degraded  transmission  quality,  by  applying  noise  and/or  reducing  signal 
strength. 

Scenarios  may  be  stopped  and  continued  from  the  point  of  interruption.  T&E  staff  can  insert 
markers  into  the  stream  of  data  to  allow  quick  identification  and  fast  forward  to  points  of  interest 
during  data  playback.  There  is  also  a  generic  slow  motion,  backup  and  fast  forward  capability. 

3.1 .5.  CCS  Data  recording 

The  system  provides  the  capability  of  recording  all  CCS  data  from  every  position  in  the  SimOps  on 
a  running  timebase.  The  timing  precision  of  this  is  unknown  at  the  present  time. 
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The  CCS  Data  window  displays  the  data  pertaining  to  the  selected  events  and  environments.  The 
CCS  Data  window  is  limited  to  200  rows.  When  the  bottom  row  is  reached,  the  first  row  is  deleted 
and  the  new  event  is  inserted  at  the  bottom  of  the  array.  CCS  Data  is  available  for  debriefing,  so 
that  events  available  for  display  in  the  CCS  data  window  can  be  played  back  later  in  a  game  mode 
called  Debrief  mode. 

This  Data  window  provides  information  on  the  following: 

•  GAME  TIME  -  Specifies  the  ORTT  game  time  at  which  the  event  occurred. 

•  (EVENT  ID)  CCS  EVENT  NAME  -  Provides  the  identification  and  name  of  the  event. 

•  DETAIL  -  Provides  specific  data  on  the  event  as  available.  The  Greenwich  Mean  Time 
(GMT)  time  for  each  event  is  shown. 

There  is  no  capability  at  present  for  adding  T&E  system  functions  to  capture  data  of  interest  that  is 
not  currently  being  recorded.  Software  to  create  such  functionality  would  have  to  be  developed 
under  contract. 

WS  Screen/input  devices 

Within  the  TCF  there  is  a  capability  of  monitoring  in  real  time  the  content  of  the  display  and  some 
user  actions.  Manual  actions  such  as  key  strokes,  QAB  selection  and  joystick  movements  are  not 
themselves  displayed  remotely,  but  their  outcomes  are. 

Communications 

Communications  at  each  position  can  be  recorded  and  time  stamped. 

Video  recording 

Two  video  cameras  provide  a  global  view  of  all  workstations  with  wide  area  coverage.  The  video 
data  are  recorded  on  separate  video  recorders;  microphones  pick  up  general  noise  and  are  recorded 
on  a  soundtrack.  Additional  video  recording  could  be  added  (e.g.  to  monitor  an  operator),  but  the 
record  would  be  independent  of  the  existing  video  recording  system  and  would  need  to  be 
subsequently  integrated  with  the  scenario  timebase. 

3.1 .6.  Data  replay  and  analysis 

A  group  debrief  facility  is  provided  in  auditorium  with  3  large  projections  screens  and 
loudspeakers  for  audio  playback.  In  addition,  data  recorded  during  a  training  session  may  be 
replayed  in  the  TCF.  The  system  provides  a  debrief  mode  which  allows  replay  of  a  simulation  at 
one  of  four  (0.5,  1,  2  or  5)  play  speeds  for  the  debrief  session  of  data  that  have  been  recorded.  A 
Debrief  Control  is  a  tool  available  to  the  training  staff  to  debrief  trainees  on  completion  of  training 
sessions.  It  uses  game  data  saved  by  the  Game  Monitoring  functions.  Three  monitors,  an  audio 
system  and  a  screen  projector  are  available  to  debrief  the  trainees.  Debrief  session  data  is 
assembled  using  the  Debrief  Control  panel.  Information  including  hardware  panels,  CCS  data, 
synthetic  environment,  Link-11,  CCS  tactical  pictures,  trainee  consoles  and  audio  recordings  can 
be  presented.  Exercise  data  may  also  be  replayed  in  the  TCF. 

Editing  tools  are  available  to  assemble  and  compile  an  audio/video  presentation  from  the  recorded 
data.  There  is  a  capability  to  playback  the  data  at  slower  or  faster  than  normal  speeds  and  to  jump 
to  specific  points  of  interest  that  have  been  tagged  either  prior  to,  or  during,  the  scenario  execution. 
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3.1.7.  Availability 

The  facility  is  currently  used  16  hours  day,  5  days  week,  through  2001/2002;  8  hours  day  are  for 
training  and  8  hours  assigned  to)  for  software  development. 

The  ORTT  must  be  booked  by  March-April  for  the  following  year's  schedule.  The  scheduling 
process  commences  in  March  and  is  completed  by  September.  Training  is  given  first  priority  and 
blocks  are  assigned  to  CFNOS  in,  March,  April,  September  and  October;  in  addition,  the  facility  is 
not  available  during  vacations  and  CFNOS  leave  periods.  Other  ORTT  priorities  are  Megaphase, 
Marlant  OTTs  (4x5days)  and  sub-team  training  The  current  usage  is  extensive  for  shakedown  and 
building  scenarios  but  it  is  expected  that  in  the  future  L-M  needs  will  be  reduced.  The  facility  is 
now  30%  overbooked.  The  air  warfare  side  is  most  congested  right  now. 

While  the  equipment  may  be  available  during  spare  time  in  Megaphase,  there  would  be  no  staff  for 
running  the  trainers;  (note  these  TCF  staff  are  needed  for  scenarios  logistics).  However,  the 
demands  for  staff  would  be  reduced  if  only  one  warfare  area  would  need  to  be  simulated.  It  would 
be  possible  to  fit  in  a  three  day  pilot  some  time  this  year;  longer  blocks  of  time  could  be  available 
next  year.  It  is  expected  that  NCOT  will  reduce  some  of  the  load  or  ORTT  when  NCOT  fully 
comes  on  line  because  it  provides  a  capability  for  training  both  basic  equipment  familiarisation 
skills  and  operational  procedures. 

Suitable  data  might  be  available  from  existing  training  courses  to  assess  the  feasibility  of  extracting 
and  analysing  relevant  elements  for  MOP  baselining. 

3.1 .8.  Support  for  scenario  and  test  development 

Excellent  detailed  documentation  is  available  that  describes  the  facility  and  how  to  build,  run  and 
replay  scenarios.  This  material  would  provide  a  basic  level  of  understanding  for  the  T&E  team  on 
how  to  implement  scenarios.  However,  the  complexity  of  the  task  means  that  a  more  efficient 
approach  will  be  to  utilise  the  existing  skilled  support  staff  in  order  to  develop  scenarios  and  part 
tasks  for  MSDF  assessment.  Such  staff  are  available  but  they  already  have  significant  ongoing 
commitments  in  building  and  running  training  scenarios. 

It  generally  takes  about  one  month  to  build  a  scenario  from  scratch.  However,  components  of 
existing  scenarios  may  be  more  readily  implemented.  Most  of  the  training  scenarios  will  be 
available  within  a  year.  Many  of  the  gaming  scenarios  are  available  now;  these  are  not  full  team, 
mission  oriented  scenarios,  but  represent  part-tasks  for  single  operators.  Given  available  ORTT 
resources,  training  could  be  provided  to  test  and  evaluation  personnel  to  develop  the  skills  for  real¬ 
time  insertion  of  data  into  existing  scenarios. 

Depending  upon  other  booking  constraints,  support  staff  could  be  made  available  to  run  the  TCF 
side  of  operations  in  any  future  use  of  the  facility  for  test  and  evaluation  purposes.  However,  data 
analysis  would  have  to  be  performed  outside  of  training  usage  hours  and  may  need  to  be 
subcontracted  to  L-M. 

3.1 .9.  Support  for  running  T&E  scenarios 

The  ORTT  infrastructure  of  hardware,  software  and  personnel  has  the  potential  to  provide  all  of 
the  required  support  for  the  running  of  T&E  scenarios.  In  particular,  skilled  personnel  are  available 
to  run  the  software  and  to  act  as  the  necessary  game  players  to  provide  the  required  dynamic  real 
time  inputs  and  responses  to  test  participants'  actions. 
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Adequate  space  is  available  in  the  TCF  to  allow  T&E  staff  to  monitor  the  scenario  events  and 
participants  actions  at  each  SimOps  workstation. 

The  configuration  of  the  system  has  limited  flexibility  but  allows  all  four  back  row  stations  to  play 
the  same  role  (e.g.  CO,  ORO,  SWC,  ASWC)  to  permit  multiple  test  participants  doing  the  same 
pre-packaged  tasks,  although  the  upper  limit  for  air  functions  would  be  three  simultaneous 
workstations. 

3.1.10.  Support  for  data  analysis 

The  TCF  provides  much  of  the  required  support  for  data  reduction  and  analysis.  However,  this 
work  is  carried  out  currently  under  contract  by  L-M,  hence  there  is  not  at  present  a  toolsuite 
available  for  the  T&E  team  to  independently  reduce  and  analyse  the  data.  Further,  the  analysis 
must  be  conducted  on-site.  Provision  of  a  tool-suite  to  permit  off-site  data  reduction,  analysis  and 
presentation  would  significantly  enhance  the  T&E  capability  of  this  simulator. 

3.1.11.  Summary  of  advantages  and  disadvantages 

Advantages 

•  Comprehensive  scenarios  that  encompass  the  full  spectrum  of  Ops  personnel  functions 
and  tasks. 

•  High  degree  of  face  validity  in  terms  of  equipment,  layout  and  operational  conditions. 

•  Available  tools,  expertise  and  knowledge  to  build  scenarios. 

•  Scenarios  run  with  a  high  degree  of  fidelity  to  actual  operational  environments 

•  Pre-packaged  scenarios  are  under  development  and  will  be  available  shortly. 

•  Supporting  players  available  to  play  the  necessary  background  roles  to  feed  data  to 
scenarios,  move  game  pieces  etc. 

•  Capability  to  insert  T&E  real-time  data  into  scenarios. 

•  Allows  for  multiple  players  in  team  tasks  or  multiple  players  (limited)  on  a  single  task. 

•  Real  time  logging  of  user  actions,  contents  of  screens,  messages. 

•  Real  time  monitoring  of  user  actions,  screens,  messages  and  space  for  observers  to  do 
some  real  time  data  capture  in  TCF. 

•  General  audio  and  video  recording  of  overall  SimOps 

•  Two  SimOps  chambers  available 

•  Excellent  capability  for  data  playback  for  individual  and  all  workstations 

•  Potential  capability  to  integrate  MSDF  functions  into  existing  C2  software 


Disadvantages 

•  Facility  is  in  demand  and  access  time  is  scarce.  T&E  would  need  to  receive  priority 
support  from  high  levels  of  Command  Authority  in  order  to  jump  the  queue. 

•  A  large  personnel  infrastructure  is  required  to  develop  and  run  scenarios. 

•  Analysis  of  scenario  data  will  require  sub-contracting  some  initial  data  reduction  and 
compilation  to  L-M. 
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•  Additional  video  and  audio  data  would  have  to  be  manually  integrated  into  existing 
recording  capability. 

•  Reduced  data  may  not  be  taken  off-site  for  subsequent  analysis. 

•  Any  additional  custom  software  modules  for  T&E  purposes  would  need  to  be 
contracted  to  L-M. 

•  No  recording  of  individual  face  to  face  conversations  without  additional  microphones. 

3.1.12.  Conclusions 

The  ORTT  provides  all  the  technical  requirements  to  run  a  range  of  scenarios  in  scope  and 
complexity  with  a  full  or  partial  combat  team.  The  support  infrastructure  provides  the  necessary 
capability  to  allow  most  of  the  anticipated  T&E  requirements  to  be  accomplished.  While  there  are 
some  minor  areas  where  additional  capability  might  have  been  required  (video  and  audio  recording 
of  individual  team  members),  the  major  disadvantage  in  using  the  facility  relates  to  its  lack  of 
availability,  large  infrastructure  requirement  and  limited  re-configuration  ability. 

With  respect  to  availability,  unless  T&E  activities  associated  with  COMDAT  are  given  priority 
over  or  can  be  combined  with  training  functions,  then  there  will  be  only  limited  access  to  the 
facility  over  the  next  two  year  span.  The  best  that  might  be  expected  could  be  two  blocks  of  3-4 
days  each,  according  to  current  and  anticipated  bookings. 

In  providing  this  high  level  of  simulation  capability,  the  ORTT  requires  an  infrastructure  that  is 
very  comprehensive  and  labour  intensive  in  support  of  scenario  development  and  execution.  Even 
if  pre-packaged  scenarios  could  be  used  for  COMDAT  T&E  purposes  all  of  the  required  personnel 
would  still  have  to  be  brought  together  to  support  the  execution  of  each  test  trial.  The  potential  for 
conflicting  demands  on  the  services  of  these  individuals  will  further  constrain  when  T&E  activities 
could  be  scheduled  for  the  ORTT. 

Therefore,  taking  into  account  the  availability  of  the  ORTT,  its  infrastructure  and  overhead 
requirements,  our  conclusion  is  that  it  should  only  be  used  for  those  COMDAT  T&E  purposes  that 
will  need  comprehensive  scenarios  with  multiple  interacting  combat  team  players.  This  might 
involve  whole  team  functions  at  the  warfare  section  level,  or  team  tasks  involving  the  CO,  ORO 
and  Warfare  Directors. 

We  consider  that  it  would  be  inappropriate  use  of  both  T&E  and  Navy  limited  resources  to  use  the 
facility  for  part  task  and  multiple  task  (as  opposed  mission  oriented  functions)  T&E  activities,  in 
the  light  of  the  overall  level  of  effort,  planning  and  management  required  to  set-up  the  required 
tasks  and  collect  and  analyse  the  data. 

While  actual  requirements  still  have  to  be  calculated,  a  reasonable  estimate  might  be,  at  least,  one 
week  for  preparation  and  pilot  rehearsal  and  one  week  for  data  capture,  each  year,  plus  some 
software  development  support  (more  in  early  years).  The  schedule  for  T&E  depends  on  COMDAT 
program  but  could  be  as  early  as  Fall  2002.  Preparatory  development  of  data  capture  software  and 
evaluation  scenarios  could  start  earlier. 

3.2.  NCOT 

The  information  presented  below  is  based  on  a  discussion  with  a  McDonald  Dettweiler  (MD) 
technical  representative  who  has  the  local  responsibility  for  the  implementation  and  development 
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of  NCOT,  a  CD  containing  screen  dumps  of  the  major  NCOT  functions  and  extracts  from  system 
documentation. 

3.2.1.  Overall  Description 

Examples  of  an  NCOT  workstation  and  a  grouping  of  workstations  are  shown  below.  Note  that  the 
left  screen  in  the  work  station  (Figure  6)  simulates  the  CCS  screen  and  the  right  hand  screen 
simulates  the  control  panel  of  choice.  The  keyboard  (etc)  is  only  an  approximate  simulation  of  the 
"look  and  feel"  of  the  actual  CCS  workstation. 


Figure  6:  Sample  NCOT  Workstations  simulating  the  RT1  position  in 
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Figure  7:  A  grouping  of  NCOT  Workstations 

The  NCOT  has  been  designed  to  provide  initial  training  and  refresher  training  to  students.  The 
simulations  provide  a  training  environment  for  equipment  operators  to  practice  the  integration  of 
knowledge  and  skills  to  detect,  analyse,  classify,  track,  report,  manage  data,  and  prosecute  contacts 
as  would  be  required  under  an  equivalent  operational  environment. 

Multiple  configurations  are  possible  with  NCOT.  Workstations  (W/Ss)  can  operate  independently 
for  individual  action/reaction,  or  they  can  be  grouped  together  so  that  students  can  participate  in  the 
same  game.  Although  the  NCOT  flexibility  enables  it  to  be  configured  in  ways  different  from  an 
actual  Ops  Room,  (e.g.,  assigning  multiple  Track  Supervisor  (TS)  stations  to  the  same  ownship), 
the  interaction  of  components  in  such  a  configuration  is  not  predictable. 

The  NCOT  facility  at  CFNOS  is  currently  in  the  process  of  being  implemented  and  its  full  range  of 
functionality  is  not  yet  in  place.  There  are  38  workstations  (2  classrooms  each  with  16  operator 
positions,  1  instructor  position  and  2  developmental  positions,  which  can  be  used  as  any  operator 
position  if  desired.  There  is  one  briefing  room  with  2  big  screens.  A  further  nine  workstations  have 
been  installed  in  TRINITY  (Stadacona)  and  there  are  a  further  48  workstations  on  the  West  Coast 
configured  as  four  classrooms,  but  with  no  briefing  room  and  local  MD  support. 

The  workstations  are  configured  in  two  rows  that,  as  presently  laid  out,  loosely  resemble  the 
standard  Ops  Room  format  (see  Figure  7).  Some  workstations  have  been  grouped  together  as  twos 
or  fours  to  allow  for  computer-based  training  and  other  instructional  purposes.  The  furniture  and 
fittings  of  the  facility  have  little  in  common  with  the  ruggedised  equipment  environment  of  an  Ops 
Room. 

Each  workstation  has  a  keyboard,  joystick,  rollball  and  mouse  for  data  entry/action  purposes.  This 
hardware  is  a  standard  commercial  workstation  product  and  differs  physically  from  similar  devices 
found  in  the  Ops  Room,  although  the  layout  is  approximately  the  same.  Hence  the  look  and  feel  of 
input  devices  has  somewhat  lower  fidelity  to  the  actual  operating  environment. 

Coloured  CRT  monitors  can  simulate  any  Ops  Room  CCS  display,  communication  or  other  panel 
(e.g.  weapons  control).  Thus,  again  there  is  lower  fidelity  in  the  look  and  feel  compared  with  the 
Ops  Room. 

The  capability  to  monitor  what  events  and  actions  are  occurring  on  an  NCOT  workstation  is 
provided  by  instructor  workstations  alongside  the  trainee's  workstation,  as  well  as  remote 
monitoring  in  a  room  next  to  the  main  test  facility.  The  plan  is  to  fully  equip  this  adjacent  room  in 
order  to  enhance  the  remote  monitoring/scenario  interaction  capability. 

NCOT  software  runs  on  a  variety  of  HP  workstations  using  X  Windows  and  Motif,  VAPS,  Gain, 
Windows  95  and  Simphonics  V+.  This  means  that  it  has  the  potential  to  be  run  on  workstations 
located  outside  the  facility.  However,  there  is  currently  no  capability  to  transfer  NCOT  data  from 
one  NCOT  to  another,  or  between  NCOT  and  other  PCs.  Part  of  this  is  a  security  concern  and  part 
is  a  hardware  limitation.  Transferability  of  the  type  desired  for  T&E  purposes  does  not  appear  to  be 
a  priority  for  the  Navy,  and  so  probably  will  not  be  implemented  in  the  near  future,  if  ever.  Hence, 
all  T&E  work  would  have  to  be  done  at  one  of  the  NCOT  facilities. 

NCOT  software  is  currently  based  on  CCS  Version  4  which  is  a  couple  of  versions  behind  what  the 
Navy  is  currently  using..  The  plan  eventually  is  to  have  it  about  6  months  behind  what  the  ships 
have  and  plans  are  in  place  to  update  this  to  Version  6  within  the  next  several  months. 
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3.2.2.  Simulation  capability 

The  functionality  of  all  of  the  combat  team  workstations  can  be  simulated  using  MD  proprietary 
SIMWARE,  including  CANEWS,  CANTASS  and  HMS.  There  is  no  workstation  specifically 
configured  for  bridge  functions,  but  simulated  information  from  the  bridge  or  a  helo  can  be  fed  into 
a  scenario.  Unlike  the  ORTT  which  uses  identical  hardware  to  an  actual  Ops  Room,  in  NCOT  the 
CCS  and  other  operational  systems  displays,  as  well  as  input  devices  such  as  comm  panels,  are  all 
simulated  graphically  on  a  CRT.  There  is  no  capability  to  simulate  GCCS  and  the  facility  is  not 
equipped  with  ancillary  information  sources  such  as  stateboards,  charts,  TacPacs  etc,  although  in 
principle  there  is  ample  space  available  to  provide  these  in  the  future. 

The  full  range  of  network  audio  comms  found  in  an  Ops  Room  can  be  simulated.  Workstations  can 
be  flexibly  configured  to  act  as  a  single  Ops  Room  workstation  (e.g.  the  ASWC)  or  grouped 
together  in  which  each  workstation  has  the  same  role  or  functionality,  i.e.  there  could  be  four  or 
five  workstations  that  are  fed  the  same  scenario  and  tasks,  and  test  participants  could  all  perform 
the  same  tasks  in  parallel.  Further,  workstations  can  be  configured  to  allow  small  teams  to  function 
and  interact. 

SIMWARE  provides  the  Simulation  Trainer  Control  functionality,  the  Master  Simulator,  dynamic 
game  control,  game  scenario  definition,  entity  definition,  and  environment  definition.  SIMWARE 
also  provides  sensor  modelling  for  HALIFAX  and  IROQUOIS  class  sensors,  systems.  Combat 
Control  System  (CCS)  modelling  and  simulation,  and  the  definition  of  Student  Operator  roles. 

Conceptually,  the  system  appears  to  be  planned  to  accommodate  functionality  that  would  support 
the  full  range  of  Ops  Room  functions  to  be  performed  in  a  co-ordinated  manner  in  mission  oriented 
scenarios.  At  present,  its  capability  seems  to  be  focussed  on  providing  training  simulation  for  tasks 
and  part  tasks  at  the  individual  operator  level,  independent  of  interactions  with  the  full  combat 
team.  This  training  is  oriented  towards  equipment  familiarisation  and  performing  basic  tasks. 

NOTE:  At  present,  there  is  only  software  available  to  simulate  the  front  row  team;  software  to 
support  the  back  row  is  scheduled  for  delivery  on  July  1/2001. 

3.2.3.  Scenario  Development 

It  appears  that  all  existing  scenarios  are  being  developed  by  MD  off  site  from  NCOT,  and  there  is 
no  Navy  capability  to  do  this  at  present.  Hence,  future  T&E  scenario  requirements  may  have  to  be 
separately  contracted  to  MD.  However,  it  appears  possible  that  T&E  personnel  with  a  senior  Ops 
Room  background  could  be  trained  by  MD  in  a  matter  of  days  to  develop  specific  T&E  scenarios. 

There  has  been  some  development  of  scenarios  but,  at  present,  these  are  somewhat  limited  in  scope 
and  comprise  tasks  and  sub-tasks  for  individual  combat  team  members  rather  than  complete 
missions  in  which  full  combat  teams  interact  with  one  another  with  real  time  support  from  game 
controllers  and  role  players.  These  scenarios  comprise  basic  training  elements  for  the  front  row 
team  only. 

Limited  documentation  exists  at  present  for  existing  scenarios  or  those  under  development. 
Currently,  scenarios  typically  take  a  day  or  two  to  build,  for  example  a  two  hour  scenario  centred 
on  one  combat  team  function,  would  take  between  an  hour  and  half  a  day,  if  based  on  existing 
event  database.  It  is  the  intention  that  in  the  final  system  that  the  Instructor  Station  (IS)  will  have 
the  capability  to  develop  new  scenarios  off-line,  configure  and  reconfigure  workstations  in  the 
classroom,  and  brief/debrief  the  scenario  to  students. 
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A  Scenario  Builder  application  runs  independently  of  the  main  Instructor  Station  Tactical  Display. 
This  module  provides  a  powerful  development  tool  for  the  creation  and  editing  of  scenarios.  The 
Landmass  Generator  is  also  a  stand-alone  module  that  provides  the  operator  with  the  ability  to 
create  user-defined  landmasses  for  use  in  scenarios.  An  Entity  Library  Editor  (ELE)  is  available  to 
developers  for  use  in  editing  entity  characteristics  for  permanent  information  changes  to  the  entity 
class  description. 

The  heart  of  the  IS  is  the  Tactical  Display,  which  is  the  main  user  interface  for  the  Instructor  during 
the  running  of  the  game.  The  Tactical  Display  provides  a  view  of  the  game  area  and  control  over 
each  entity  in  a  game  to  the  Instructor.  In  addition,  options  on  the  Tactical  Display  menu  allow  the 
Instructor  to  modify  certain  environmental  factors  in  a  game. 

There  would  also  be  an  additional  overhead  associated  with  running  T&E  scenarios  with  respect  to 
configuring  each  workstation  for  each  test  session.  This  would  involve  setting  up  the  radars  and 
other  functions  like  fire  control. 

3.2.4.  Real-time  control  and  monitoring  of  scenarios 

SIMWARE  running  on  the  Instructor  Station  provides  the  capability  to  control  the  game(s),  control 
the  vehicles/entities  within  the  game,  identify  automatic  student  assessment  parameters,  change 
certain  parameters  during  the  game,  etc. 

The  Instructor  Station  is  made  up  of  several  software  components  that  provide  graphical  interface 
for  the  Instructors  as  well  as  background  calculations  for  the  running  of  the  scenarios. 

The  simulation  also  provides  the  following  capabilities: 

•  Radar.  Radar  simulation  is  designed  to  calculate  the  target  detectability  based  on 
effective  range  of  a  particular  radar  during  each  iteration  of  the  radar  calculations. 
When  an  entity  or  landmass  point  meets  detection  criteria,  the  appropriate  information 
is  passed  through  the  internal  network  to  the  Graphicsware  process  running  the  Student 
Station  for  display.  The  Radar  Sim  includes  an  Identification  Friend  or  Foe  (IFF) 
component.  This  capability  will  be  important  for  MSDF  T&E. 

•  Sonar.  Sonar  simulation  performs  sonar  acoustics  simulation  for  towed  array,  hull 
mounted  sonar,  and  sonobuoy  processing  systems  emulations. 

•  Electronic  Warfare  (EW).  EW  simulation  includes  both  electronic  warfare  (EW) 
components  and  passive  and  active  countermeasures.  The  EW  simulation  has 
components  for  Canadian  Electronic  Warfare  System  (CANEWS),  SRD-502, 
Reprogrammable  Advanced  Multimode  Shipboard  ECM  System  (RAMSES),  and 
SHIELD  processing. 

•  CCS.  CCS  simulation  provides  emulation  of  10  HALIFAX  Class  control  console 
operator  roles:  Assistant  Sensor  Weapons  Controller  (ASWC),  Electronic  Warfare 
Supervisor  (EWS),  Operations  Room  Officer  (ORO),  RT1,  RT2,  SAC,  SCS,  SPS, 
Sensor  Weapons  Controller  (SWC),  and  CCS. 

•  Weapon  Control  System  (WCS).  WCS  simulation  contains  components  that  provide 
simulation  for  Separate  Tracking  and  Illumination  Radar  (STIR),  Close  In  Weapon 
System  (CIWS),  Weapons  Control  Panel  (WCP),  and  HARPOON,  as  well  as  the 
launching  of  acoustic  countermeasures,  flares,  drones,  and  chaff. 

•  IFF.  IFF  simulation  supports  the  functionality  of  IFF  panels  and  supply  responses  to 
simulated  IFF  interrogation  queries.. 


Page  24 


Review  of  T&E  Environments  for  COMDAT  MOPs 


Humansysrems  Incorporated 


P516804.PDF  [Page:  36  of  100] 


•  Link-11.  Link-1 1  Remote  Control  Panel  (RCP)  simulation  is  a  part  of  the  TS  operator 
position.  It  will  not  simulate  Link-1 1  transmission  times,  nor  the  functionality  of  the 
Net  Control  Station.  Basic  RCP  controls  and  indicators  allow  the  TS  operator  to 
perform  the  fundamental  steps  in  order  to  enable  receipt  and  transmission  of  Link- 1 1 
messages  from  his/her  ownship. 

•  Aircraft  Control  Panel.  Aircraft  Control  Panel  simulation  provides  trainer-specific 
capabilities  which  enable  students  and  instructors  to  control  the  assets  of  HALIFAX 
Class  entities  once  they  are  launched.  It  enables  control  of  course,  speed,  altitude  and 
launching  of  sonobuoys,  missiles  and  torpedoes. 

•  Radiation  Control  Monitor  (RClvD.  RCM  simulation  provides  support  for  the  RCM 
panel  and  its  associated  controls  by  passing  messages  to  the  appropriate  station. 

•  TACAN.  The  TACAN  simulation  provides  homing  beacons  for  the  air  assets  of 
ownship.  This  is  simulated  by  passing  messages  to  the  appropriate  stations. 

Each  student  role  simulation  is  associated  with  a  particular  Ops  Room  function  selected  by  the 
instructor.  All  sensor-based  simulation  is  concerned  with  the  detectability  of  each  entity  that  is 
defined  in  the  scenario.  It  must  evaluate  the  position  of  each  entity  with  respect  to  ownship  at 
regular  intervals.  The  position-keeping  engine  for  the  game  updates  the  position  of  each  entity 
once  per  second.  Entities  move  according  to  motion  characteristics  defined  in  the  scenario,  but 
also  respond  to  instructor  ordered  changes  in  course,  speed,  altitude  or  depth.  When  any  of  these 
are  modified,  the  actual  motion  will  change  according  to  its  characteristics  defined  in  the  entity 
library.  The  motion  of  entities  can  be  paused  and  resumed  by  the  instructor. 

The  instructor  may  control  and  vary  the  environmental  conditions  of  the  game.  Each  of  the 
environmental  conditions  effect  student  role  simulations  in  a  variety  of  ways: 

•  Sea  State  -  Sea  State  effects  the  radar  clutter  levels  of  radar  simulations  and  the 
ambient  noise  level  for  passive  sonar  simulations. 

•  Wind  -  Wind  direction  and  speed  effects  the  motion  of  entities  in  the  position-keeping 
engine  of  the  game. 

•  Atmospheric  Reffactivity  -  The  type  of  atmospheric  environment  effects  the  radar 
horizon  for  radar  and  EW  simulations. 

•  Distant  Shipping  Noise  -  Omni  directional  distant  shipping  noise  effects  the 
propagation  loss  curves  for  passive  sonar  simulations. 

•  Ocean  Model  -  Ocean  model  defines  different  sets  of  propagation  loss  curves  for 
passive  and  active  sonar  simulations. 

•  Clouds  -  radar  video  and  detectability  of  targets 

3.2.4. 1.  General  Communications  capability 

Voice  over 

As  a  student  or  instructor  speaks  into  the  microphone,  the  voice  digitized  and  broadcast  to  all  other 
stations.  Each  station  in  the  trainer  can  receive  all  voice  communications  but  they  are  locally 
filtered  based  on  rules  specified  in  the  communications  plan  for  the  scenario.  The  instructor  prior 
to  starting  a  game  assigns  the  communications  plan  to  the  station.  When  stations  are  not  included 
into  an  active  game,  they  are  provided  with  a  default  communications  plan. 
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Communications  Plan 

A  communication  plan  defines  fast  action  buttons  on  the  communications  terminal.  Each  fast 
action  button  has  an  on/off  state  and  is  defined  as  making  a  connection  to  an  ownship  intercom  or 
radio.  If  a  radio  is  defined,  then  the  attributes  of  the  radio  must  also  be  specified.  Intercom  voice 
communications  are  within  ownship  and  radio  voice  communications  between  ships.  Intercom 
voice  communication  is  full  duplex  and  radio  voice  communication  is  half-duplex.  Within  a  game, 
if  multiple  stations  assigned  to  the  same  ownship  select  the  same  intercom,  then  voice  activity  is 
heard  by  all  connected.  Also,  if  stations  assigned  to  different  ownships  select  the  same  radio,  then 
voice  activity  is  heard  only  if  the  ships  are  within  radio  range  of  each  other. 

Point-To-Point  Communications 

Each  station  can  call  any  other  station  in  the  trainer.  When  the  station  is  a  part  of  a  game,  each 
station  in  the  game  can  call  only  those  stations  that  are  a  part  of  the  same  game  and  assigned  to  the 
same  ownship.  Experienced  personnel  may  be  required  to  provide  appropriate  communications 
input  and  responses  appropriate  to  the  scenario  context.  In  the  case  of  more  simple,  routine 
scenarios  where  communications  are  more  predictable,  a  less  experienced  person  may  provide  the 
necessary  communications  by  following  a  script. 
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Personal  Address  System 

The  instructor  is  provided  with  a  public  address  system.  The  following  PA  capabilities  are 
allowed: 

•  Address  all  stations  in  the  trainer 

•  Address  all  stations  in  the  classroom 

•  Address  all  stations  grouped  into  a  single  game  within  a  classroom 

•  Address  all  stations  assigned  to  the  same  ownship  within  a  game 

Jamming  Radios 

If  a  radio  is  defined  in  the  communications  plans  used  within  a  game  the  instructor  can  jam  it. 
'Jamming'  a  radio  consists  of  the  instructor  selecting  a  radio  jamming  WAV  file  from  a  set  list  and 
applying  it  to  one  of  the  radios  in  the  game.  The  instructor  also  selects  the  volume  of  the  WAV 
file.  Jamming  a  radio  can  be  established  at  the  beginning  of  the  game  or  as  the  game  progresses 

3.2. 4.2.  Student  Monitoring  Tools 

The  instructors  set  of  tools  includes  software  that  allows  for  monitoring  the  voice  communications 
of  the  students  in  the  classroom.  Also,  each  individual  student  desktop  activity  may  be  monitored 
using  a  separate  software  package. 

Communications 

The  GPRT  audio  monitoring  software  uses  the  classroom  configuration  settings  to  read  and  the 
communications  plans  assigned  to  each  station  in  the  classroom.  It  builds  a  list  of  all  possible 
point-to-point  connections  that  can  be  made  by  the  instructor.  It  also  builds  a  list  of  all  the  possible 
public  address  options  available  to  the  instructor.  As  the  game  progresses  and  each  student 
selects/deselects  intercoms  and  radios  for  voice  communications,  this  software  keeps  an  up-to-date 
list  of  all  active  selections  by  students.  The  instructor  may  choose  any  active  intercom,  radio  or 
point-to-point  call  for  monitoring.  The  instructors'  presence  is  not  noticeable  to  the  students  unless 
the  instructor  depresses  the  microphone  to  speak.  This  allows  the  instructor  to  monitor  students  for 
appropriate  voice  communications  and  intervene  only  when  deemed  necessary. 

Workstation  monitoring 

The  GPRT  desktop  monitoring  software  allows  the  instructor  to  choose  any  station  in  the 
classroom  and  bring  the  desktop  activity  of  that  station  to  the  desktop  of  the  instructor.  This  type 
of  monitoring  can  be  done  inside  or  outside  the  context  of  the  game  and  only  requires  that  the 
student  login  at  the  station  to  be  complete.  The  instructor  also  has  the  ability  to  use  the  mouse  at 
the  instructor  station  to  point  to  areas  of  the  student  desktop.  In  this  case,  both  the  instructor  and 
student  see  the  mouse  as  it  moves. 

More  generally  NCOT  provides  the  following  functionality: 

•  Scenario  events  may  be  controlled  either  through  pre-programmed  scenario  scripts  and 
associated  underlying  databases  or  by  real-time  dynamic  input  from  game  controllers. 
Depending  upon  the  degree  of  complexity  and  or  uncertainty  of  the  scenario, 
controllers  may  be  either  personnel  with  the  appropriate  military  knowledge,  or  for 
simple  scenarios,  non-military  personnel  who  are  trained  to  follow  a  script. 
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•  Audio  and  text  messages  that  are  not  already  scripted  may  be  sent  to  any  workstation 
from  a  sim  control  workstation.  The  GPRT  provides  the  ability  to  play  sounds  in  the 
headset  of  each  station  while  a  game  is  running.  It  also  provides  for  regulated  voice 
communications  between  stations  along  with  a  public  address  system  for  the  instructor. 
Radios  that  are  defined  in  the  game  may  be  jammed  at  the  discretion  of  the  instructor. 

•  Any  sim  workstation  can  be  monitored  on  a  second  workstation,  including  screen 
content,  pointers,  cursors,  actions  on  simulated  input  keypads 

•  Printouts  can  be  taken  of  screen  snapshots 

A  scenario  may  be  frozen  at  any  point  in  time,  with  all  screen  display  information  intact,  to 
accommodate  some  aspect  of  T&E  assessment.  The  scenario  may  then  be  re-started  and  will  pick 
up  from  the  point  at  which  it  was  halted. 

At  present  there  is  no  separate  simulation  suite  that  corresponds  to  the  TCF  in  the  ORTT,  although 
an  adjacent  room  is  available  for  this  purpose  should  it  be  needed. 

3.2.4.3.  Logistical  support  for  running  scenarios 

There  is  limited  personnel  support  for  running  scenarios  at  present.  The  MD  technical 
representative  is  the  only  person  with  the  current  skills  to  build,  run  and  control  scenarios  from  the 
instructor  position.  One  Navy  representative  indicated  that  he  thought  that  T&E  activities  should 
be  as  self-sufficient  as  possible  and  drawing  minimally  on  Navy  resources.  This  may  present  some 
difficulties  for  more  complex  scenarios  involving  the  CO,  ORO  and  warfare  directors,  as  players 
would  be  needed  to  play  the  roles  of  lower  level  team  members  in  order  to  provide  the  required 
data.  Similarly,  one  would  need  players  to  provide  data  and  communication  from  the  usual  sources 
of  outside  of  the  Ops  Room. 

3.2.5.  Data  recording 

All  keystrokes,  screen  navigation,  actions  on  comm  buttons  and  panels,  contents  of  CCS  display 
and  network  voice  comms  can  be  captured  in  software  along  a  running  timebase.  The  temporal 
resolution  was  stated  to  be  less  than  1  second  but  this  seems  to  be  an  estimate  rather  than  a  known 
capability.  Non-network  voice  comms  would  require  additional  individual  microphones  and 
standalone  recorders.  These  would  not  be  co-ordinated  on  the  timeline  with  the  data  collected  by 
software  and  hence  would  need  to  be  integrated  after  the  fact.  There  is  no  built-in  video  recording 
capability  at  present,  however,  we  were  advised  that  there  is  ongoing  development  of  the  recording 
capability. 

The  capacity  of  the  record  capability  is  2  gigabytes,  but  there  is  no  experience  yet  to  determine 
what  this  means  in  terms  of  events  and  time  span. 

The  GPRT  desktop  record  and  playback  software  allows  the  instructor  to  record  and  group  of 
stations  together.  This  recording  includes  both  the  desktop  and  all  voice  communications  and 
sounds  associated  with  a  station.  The  instructor  can  not  only  select  stations  used  by  students  for 
recording,  but  also  the  station  in  use  by  the  primary  instructor.  This  allows  for  a  complete  review 
of  all  aspects  of  the  game  on  playback. 

When  recording  has  been  stopped,  entries  are  made  into  a  recorded  data  catalogue  for  the 
classroom.  This  catalog  is  organised  by  station  and  contains  a  description  of  each  of  the  recording 
sessions  along  with  the  filenames  that  were  created  during  recording.  For  playback,  the  instructor 
selects  the  desired  station  or  group  of  stations,  selects  the  appropriate  catalogue  entries  and  initiates 
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playback.  Playback  occurs  on  each  station  until  either  the  playback  completes  or  the  instructor 
stops  the  playback. 

3.2.6.  Data  replay  and  analysis 

Data  collected  during  the  scenarios  can  be  played  back  at  real  speed,  or  at  1/2,  2x,  4x,  8x  or  16x 
regular  speed.  Data  replay  must  always  be  started  from  the  commencement  of  the  scenario. 

Hence,  if  one  were  interested  in  a  point  75  minutes  into  a  scenario,  one  would  fast  forward  at  16x 
speed  to  the  75  minute  mark.  If  subsequently,  one  were  to  then  want  to  go  the  55  minute  mark,  the 
record  would  have  to  be  rewound  to  the  start  and  fast  forward  again  implemented.  The  playback 
cannot  be  stopped  without  losing  data  from  the  displays. 

There  is  no  existing  capability  to  analyse  or  reduce  data;  this  would  need  to  be  developed  by  MD. 

There  is  ongoing  software  development  to  enhance  the  data  replay  capability. 

The  software  to  provide  replay  is  not  yet  in  place  and  is  expected  for  delivery  on  July  1.  This  will 
allow  replay  for  one  operator  position  only  plus  ground  truth.  More  comprehensive  playback 
capability  has  been  requested  by  the  Navy,  but  according  to  one  estimate,  implementation  is  about 
one  year  away. 

3.2.7.  Availability 

It  is  expected  that  once  fully  functional,  NCOT  will  have  similar  demands  for  access  as  the  ORTT. 
Although  at  present  Navy  demand  is  currently  lower  than  for  ORTT  but  this  is  offset  by  a 
significant  usage  by  MD  for  system  development.  It  is  likely  that  the  facility  will  offload  some  of 
the  part-task  training  from  the  ORTT  in  areas  such  as  equipment  familiarisation  and  training  in 
basic  operator  tasks  and  sub-team  procedural  training. 

3.2.8.  Support  for  scenario  and  test  development 

At  present  there  is  no  infrastructure  in  place  to  provide  this.  All  scenario  development  is  carried 
out  by  the  technical  representative  of  MD  at  CFNOS,  although,  as  we  have  indicated  above,  it  may 
be  possible  to  train  an  HSI®  team  member  in  this  skill.  In  practice,  this  would  mean  that  the  T&E 
team  would  have  to  provide  the  scenario  script  in  detail,  possibly  taking  advantage  of  the  existing 
database  of  gamepieces  and  events.  It  is  possible  that  some  the  small-scale  scenarios  that  have 
been  created  to  date  for  computer  based  training  may  be  of  assistance  in  the  creation  of  T&E  tasks 
and  part-tasks. 

3.2.9.  Support  for  running  T&E  Scenarios 

At  present  there  is  no  specific  infrastructure  in  place  to  provide  this.  However,  in  principle  similar 
staff  support  to  that  found  in  the  ORTT  as  game  controllers  and  players  could  be  provided.  Access 
to  such  support  would  have  the  same  constraints  as  for  the  ORTT.  The  overall  level  of  support 
required  to  run  NCOT  scenarios  is  significantly  less  than  for  the  ORTT.  In  fact,  one  of  the  existing 
support  personnel  indicated  that  he  could  do  it  alone  provided  there  were  players  available  to 
provide  external  or  outstation  input. 

3.2.10.  Support  for  data  analysis 

Data  captured  during  scenarios  may  be  played  back  at  normal,  accelerated  or  slow  speed.  Special 
purpose  software  would  need  to  be  developed  to  automatically  capture  and  reduce  specific  data  of 
interest  for  T&E  purposes.  Special  arrangement  would  need  to  be  made  to  analyse  data  off  site  and 
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a  specially  configured  workstation  would  be  required  locally  to  accomplish  this.  Further, 
negotiations  with  MD  would  be  required  to  address  commercial  proprietary  issues.  At  present 
there  is  no  capability  for  reducing  data  and  exporting  it  in  a  format  that  could  be  used  by  COTS 
analysis  products  (e.g.  Excel,  SPSS). 

3.2.11.  Summary  of  advantages  and  disadvantages 

Advantages 

•  The  potential  exists  to  build  comprehensive  scenarios  that  encompass  the  full  spectrum 
of  individual  Ops  personnel  functions  and  tasks. 

•  There  is  moderate  face  validity  to  the  Ops  Room  in  terms  of  equipment,  layout  to 
operational  conditions. 

•  Available  tools,  expertise  and  knowledge  to  build  scenarios. 

•  Pre-packaged  scenarios  are  under  development  and  will  be  available  shortly  mainly  for 
single  operator  tasks. 

•  Capability  to  insert  T&E  real-time  data  into  scenarios. 

•  A  high  level  of  flexibility  in  terms  of  configuring  the  T&E  environment  to  allow  for 
multiple  players  in  small  team  tasks  or  multiple  players  on  a  single  task. 

•  Could  be  readily  configured  to  allow  for  a  situation  in  which  one  controller  on  an 
adjacent  workstation  could  control  and  interact  with  a  number  of  workstations  all 
playing  the  same  role.  Hence,  there  are  personnel,  time  and  cost  savings  over  running 
single  subjects  in  single  roles. 

•  Real  time  logging  of  user  actions,  content  of  screens,  messages. 

•  Real  time  monitoring  of  user  actions,  screens,  messages. 

•  Capability  for  data  playback  for  individual  and  all  workstations  (the  latter  in  the 
future). 

•  Facility  may  be  in  less  demand  than  the  ORTT. 

•  Lower  level  of  personnel  infrastructure  required  to  run  scenarios,  compared  with 
ORTT. 

•  Potential  capability  to  integrate  MSDF  functions  into  existing  C2  software. 

Disadvantages 

•  Currently  the  facility  is  in  demand  and  access  time  is  limited.  T&E  would  need  to 
receive  priority  support  from  high  levels  of  Command  in  order  to  jump  the  queue. 

•  Analysis  of  scenario  data  will  require  sub-contracting  some  of  the  initial  data  reduction 
and  compilation  to  MD. 

•  Look,  feel  and  location  of  input  devices  somewhat  dissimilar  to  Ops  Room. 

•  Additional  video  and  audio  data  would  have  to  be  manually  integrated  into  existing 
recording  capability. 

•  The  development  of  software  modules  for  T&E  purposes  would  need  to  be  contracted 
to  MD  and/or  T&E  staff  would  need  to  be  trained. 

•  Existing  scenarios  are  limited  in  scope. 

•  No  recording  of  individual  face  to  face  conversations  without  additional  microphones. 
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•  Absence  of  stateboards  and  other  auxiliary  information  sources. 

•  Absence  of  dedicated  workstations  for  CANEWS,  CANT  ASS,  SPS  and  GCCS. 

•  No  existing  capability  for  back  row  team  functions  (expected  in  July  1/2001) 

•  No  proven  capability  for  supporting  functions  that  involve  teams,  either  at  the  warfare 
or  Ops  Room  levels. 

•  No  current  Navy  infrastructure  for  game  players  and  scenario  controllers. 

•  Limited  capability  to  pause  play  and  then  continue. 

3.2.12.  Conclusions 

NCOT  provides  much  of  the  required  technical  capability  to  run  a  range  of  scenarios  in  scope  and 
complexity  with  individual  operators  right  now,  or  with  small  teams  in  the  foreseeable  future. 

There  is  one  major  reservation  in  considering  using  the  facility  for  T&E  purposes  and  this  relates  to 
the  degree  of  sophistication  and  reliability  in  its  current  state  of  implementation.  NCOT  cannot  yet 
be  considered  as  a  fully  functional  T&E  environment  with  the  same  level  of  scenario  development 
as  the  ORTT,  nor  is  there  the  same  level  of  accumulated  experience  in  running  and  analysing 
scenarios.  Scenarios  are  limited  in  scope,  require  MD  support  to  run  and  are  centred  on  single 
operators  doing  relatively  simple  tasks.  Further,  much  of  what  we  have  learned  of  the  capability 
and  future  potential  is  based  upon  information  during  a  limited  interaction  with  the  MD  technical 
representative  and  from  system  documentation.  From  the  information  received  it  seems  possible 
the  facility  could  potentially  evolve  into  the  same  degree  of  functional  sophistication  as  the  ORTT. 
To  date,  much  of  that  functionality  remains  to  be  developed  and  rigorously  tested  and  evaluated. 

Notwithstanding  this  reservation,  one  major  advantage  of  NCOT  over  the  ORTT  is  the  flexibility  in 
functional  arrangement  of  the  workstations  to  allow  either  individual  players  to  play  individual 
team  roles  (as  in  the  ORTT)  but  also  to  allow  multiple  workstations  to  play  a  single  role.  Such  a 
configuration  represents  an  efficient  approach  to  data  collection  for  tasks  and  part  tasks,  which  are 
limited  in  scope,  but  allow  data  to  be  accumulated  in  from  multiple  players  in  parallel.  Further, 
there  may  be  a  much  lower  overhead  than  the  ORTT  for  conducting  T&E  functions  with  single 
operators. 

There  are  some  minor  issues  concerning  the  need  for  additional  capability  (video  and  audio 
recording  of  individual  team  members,  workstations  for  CANEWS,  CANT  ASS,  SPS). 

With  respect  to  availability,  NCOT  suffers  from  the  similar  demand  to  the  ORTT.  Although  at 
present,  it  is  primarily  used  for  training  individuals  in  system  and  function  familiarisation  using 
tasks  and  tasks  strings,  rather  than  full  scenarios  that  involve  interacting  combat  team  members 

The  existing  NCOT  infrastructure  that  we  have  observed  is  largely  a  commercial  function  provided 
by  MD  in  support  of  scenario  development  and  execution.  Any  future  software  to  support  T&E 
requirements  would  need  to  be  developed  under  contract  by  MD.  At  present,  there  appears  to  be 
very  limited  infrastructure  in  place  in  the  form  of  game  controllers  and  players  to  run  complete 
mission  scenarios. 

Existing  scenarios  are  limited  to  single  operators  doing  simple  tasks  for  basic  training.  It  is  not 
known  the  extent  to  which  scenarios  already  developed,  or  under  development,  for  the  ORTT  can 
be  ported  to  the  NCOT  environment. 

Our  overall  assessment  of  the  NCOT  facility  is  that  it  will  make  a  highly  suitable  environment  for 
collecting  MOPs  at  the  sub-task,  task  and  task  string  levels  for  single  positions  (front  row  now. 
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back  row  later  this  year).  It  appear  to  have  the  capability  for  T&E  scenarios  involving  individual 
warfare  teams,  in  group  sizes  from  two  to  four  players.  It  provides  the  required  capabilities  for 
building  test  scenarios,  controlling  their  execution  and  collecting  data.  It  offers  a  higher  degree  of 
flexibility  than  the  ORTT  to  configure  the  environment  to  maximise  data  collection  on  individual 
tasks. 

There  is  insufficient  evidence  available  at  the  moment  to  assess  how  suitable  the  facility  would  be 
for  the  execution  of  full  scenarios  with  complete  Ops  Room  teams.  We  do  have  some  concerns 
that  remain  to  be  resolved  over  the  reliability  of  the  system  and  the  fact  that  it  has  not  been  utilised 
sufficiently  to  the  point  where  we  can  be  confident  that  robust  data  can  be  collected  and  delivered 
for  analysis.  A  final  concern  is  that  all  T&E  work  would  be  highly  dependent  upon  the  commercial 
technical  support  that  would  need  to  be  provided  by  MD.  Although  it  appears  that  training  could 
be  provided  to  T&E  personnel  to  allow  them  to  build  simple  scenarios  and  run  them  independently 
of  the  Navy  but  with  the  assistance  of  MD. 

3.3.  CSTC 

3.3.1.  Overall  Description 

The  CSTC  comprises  a  suite  of  rooms  to  support  the  training  and  evaluation  of  Navy  personnel  in 
an  environment  that  has  high  face  validity  to  the  Ops  Room  in  a  HALIFAX  Class  frigate.  The 
physical  hardware  and  CCS  system  software  are  identical  to  that  found  on  a  frigate  and  the  layout 
closely  approximates  reality.  The  technical  infrastructure  that  supports  the  simulate  Ops  Room  is 
the  same  that  is  found  on  board  the  ship,  and,  hence  has  no  specific  in-built  capability  for  T&E 
purposes.  All  displays  are  fed  data  in  the  same  way  that  they  would  be  on  ship  and  all 
communications  function  in  a  similar  manner  to  their  ship  counterparts.  Data  for  scenarios  may  be 
fed  to  the  system  from  recordings  of  real  data  collected  on  missions.  Unlike  the  ORTT  and  NCOT, 
which  have  been  designed  from  the  ground  up  with  new  software  and  technology  to  support 
training  and  simulation  requirements,  the  CSTC  is  essentially  a  land-based  version  of  a  HALIFAX 
Class  Ops  Room  using  the  same  operational  equipment. 

3.3.2.  Simulation  capability 

The  functionality  of  all  of  the  combat  team  workstations  can  be  simulated  including  CANEWS, 
CANT  ASS,  HMS  and  SPS;  no  GCCS  is  available  and  simulated  data  can  be  fed  from  outside 
sources  such  as  bridge,  helo  and  Maritime  Patrol  Aircraft  (MPA).  The  full  range  of  network  audio 
comms  found  in  an  Ops  Room  can  be  simulated.  The  system  could  be  configured  to  allow 
simulation  and  data  collection  for  up  to  three  simultaneous  players  for  any  of  the  functions  of  the 
back  row  team.  Lighting  and  noise  levels  are  similar  to  those  found  on  ship.  The  facility  does  not 
have  the  capability  to  degrade  sensor  data  in  a  controllable  manner  to  mimic  actual  conditions  (e.g. 
as  is  found  with  the  ORTT  and  NCOT). 

3.3.3.  Scenario  Development 

A  number  of  scenarios  are  in  existence  to  run  in  this  facility.  They  could  be  used  in  whole  or 
modified  in  part  for  T&E  purposes. 
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3.3.4.  Real-time  control  and  monitoring  of  scenarios 

The  infrastructure  provides  a  capability  for  manual  input  of  all  simulation  events  and  hence  both 
freeze  and  tracking  probes  could  be  injected  by  the  T&E  team.  All  communications  required  for 
the  scenario  must  be  input  in  real-time  and  cannot  be  pre-recorded.  Text  messages  would  be 
manually  handled  in  a  similar  manner  to  the  Ops  Room.  Scenarios  can  be  run  to  a  precise  timeline 
but  may  also  be  stopped,  backed  up  and  re-run  under  manual  control.  T&E  data  that  are 
conditional  upon  team  members'  responses  could  be  input  in  real  time.  Experienced  personnel  are 
required  to  act  as  game  controllers  and  players  in  complex  scenarios.  For  simpler  scenarios, 
personnel  trained  to  follow  a  script  would  be  adequate. 

3.3.5.  Data  recording 

There  is  no  capability  at  present  for  audio  or  video  recording,  although  there  is  a  possibility  of 
adding  C-SAVRAS  (see  section  3.6.1)  in  the  future.  Neither  individual  keystrokes  nor  the  content 
of  the  CCS  can  be  recorded  although  adding  C-SAVRAS  would  support  the  latter  and  also  allow 
for  recording  of  all  net  based  communications.  Data  can  be  captured  using  the  "bus-sniffer"  which 
is  described  in  section  3.5.1.  Data  recording  is  limited  in  scale  such  that  for  a  full  scenario  with  all 
possible  information  captured,  only  ten  minutes  of  recording  could  be  made.  There  is  a  trade  off 
between  the  amount  of  information  collected  and  the  resulting  length  of  data  recorded.  Data  are 
not  captured  with  associated  time  tags 

3.3.6.  Data  replay  and  analysis 

Data  captured  during  scenarios  would  need  to  be  played  back  on  site.  There  is  no  existing 
capability  for  data  reduction  or  export.  Special  software  would  be  required  to  make  use  of  the 
History  Recording  function  were  it  to  be  implemented. 

3.3.7.  Availability 

The  CSTC  is  somewhat  more  available  than  the  ORTT.  It  runs  on  an  8-hour  shift  24/7  operation. 
LM  uses  it  for  eight  hours  per  day  and  CFNOS  for  a  further  eight  hours.  At  present  there  is  very 
little  available  capacity.  Booking  arrangements  and  priorities  are  the  same  as  for  the  ORTT. 

The  CSTC  is  currently  being  used  primarily  for  technician  training  and  software  development,  but 
is  being  used  occasionally  by  the  TTT  staff  for  sub-team  training,  particularly  where  there  is  a 
requirement  to  use  real  as  opposed  to  emulated/simulated  equipment.  An  example  is  when  training 
with  CANEWS  and  SHIELD  are  required. 

TTT  still  has  the  people  to  flash  up  and  run  the  CSTC,  but  these  are  the  same  people  who  would 
run  the  ORTT.  Thus,  the  more  support  required  for  T&E  to  run  and  amend  CSTC  scenarios,  the 
more  these  needs  would  be  competing  with  the  ORTT  which  would  probably  have  higher  priority. 

3.3.8.  Support  for  scenario  and  test  development 

Navy  support  personnel  could  be  made  available  to  assist  in  this  process.  Some  training  could  be 
provided  to  T&E  staff  to  allow  them  to  develop  the  necessary  skills  for  modifying  scenarios  and 
hence  allow  for  a  more  flexible  approach  to  scenario  development  that  is  less  constrained  by  the 
availability  of  Navy  SMEs.  Existing  scenarios  for  the  CSTC  are  not  being  updated,  but  there  are 
many  valid  scenarios  in  the  database 
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3.3.9.  Support  for  running  T&E  Scenarios 

Navy  support  personnel  could  be  made  available  to  assist  in  this  process. 

3.3.10.  Support  for  data  analysis 

Navy  support  personnel  could  be  made  available  to  assist  in  this  process. 

3.3.11.  Summary  of  advantages  and  disadvantages 

Advantages 

•  Can  simulate  comprehensive  scenarios  that  encompass  the  full  spectrum  of  Ops  team 
functions  and  tasks. 

•  High  degree  of  face  validity  in  terms  of  equipment  and  layout;  the  full  range  of  Ops 
Room  team  functions  can  be  simulated. 

•  Available  tools,  expertise  and  knowledge  to  build  scenarios. 

•  Scenarios  run  with  a  high  degree  of  fidelity  to  actual  operational  environments, 
however  information  (sensor  data,  communications)  may  not  be  able  to  be  degraded  to 
represent  operational  contexts. 

•  Complex  scenarios  can  be  created  and  a  database  of  these  is  available. 

•  Supporting  players  available  to  play  the  necessary  background  roles  to  feed  data  to 
scenarios,  move  game  pieces  etc. 

•  Capability  to  insert  T&E  real-time  data  into  scenarios. 

Disadvantages 

•  Facility  is  in  demand  and  access  time  is  scarce.  T&E  would  need  to  receive  priority  to 
jump  the  queue. 

•  Large  personnel  infrastructure  to  develop  and  run  scenarios. 

•  Additional  video  and  audio  data  would  have  to  be  manually  integrated  into  existing 
recording  capability. 

•  Significant  limitations  on  the  scope  and  detail  of  data  that  can  be  captured.  No  overall 
capture  of  individual  input  actions  or  CCS  content. 

•  Reduced  data  may  not  be  taken  off-site  for  subsequent  analysis. 

•  No  recording  of  individual  face  to  face  conversations  without  additional  microphones. 

•  Unknown  capability  to  integrate  MSDF  functions  into  existing  C2  software. 

3.3.12.  Conclusions 

The  CSTC  is  capable  of  generating  the  broad  base  of  complex  scenarios  that  would  be  required  for 
T&E  purposes  and  has  an  infrastructure  in  place  to  support  this.  The  CSTC  is  also  in  high  demand 
at  present,  but  its  more  extensive  operational  schedule  means  that  the  facility  may  be  more 
available  than  the  ORTT  or  NCOT  for  T&E.  However,  there  is  a  potential  problem  of  finding  the 
necessary  support  personnel  to  assist  in  the  running  of  scenarios. 

The  major  disadvantage  of  the  CSTC  concerns  its  lack  of  data  capture  capabilities.  These  are 
limited  in  scope  and  utility  compared  with  the  ORTT  and  NCOT.  Significant  additional  effort 
would  be  required  to  improve  the  existing  data  capture  functions  to  fully  support  the  specific 
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requirements  of  T&E.  Even  if  that  were  to  happen,  it  is  not  clear  that  data  could  be  collected  in 
volume  over  an  entire  scenario.  Further,  there  would  be  need  to  manually  log  all  T&E  data 
injected  into  the  scenario  and  integrate  this  manually  after  the  fact  with  data  recorded  during  the 
scenario. 

Similar  to  the  ORTT,  the  CSTC  would  probably  be  ill-used  for  running  small-scale  scenarios 
involving  task  strings,  tasks  and  part  tasks,  because  of  the  overhead  in  infrastructure  required  to  do 
this.  It  is  more  suited  to  the  collection  of  team  based  data  under  a  simulated  mission  scenario. 
However,  in  this  respect  it  falls  short  of  the  ORTT  in  terms  of  meeting  T&E  data  capture 
requirements. 


3.4.  Ops  Room  of  Ships  in  Harbour  (the  Wednesday  War) 

Within  Maritime  Operations  Group  One  (MOG  1),  located  in  Halifax,  various  operational  ships 
participate  in  occasional  co-ordinated  harbour  training  exercises  while  alongside.  Referred  to  as 
the  Wednesday  War  due  to  the  day  of  the  week  on  which  they  are  held,  these  exercises  consist  of 
simulated  tactical  warfare  scenarios  that  are  supported  by  functional  operations  rooms  within 
participating  ships  integrated  by  live  Link-1 1  and  radio  communications  connectivity. 

These  exercises  are  not  conducted  every  Wednesday,  but  rather  at  a  frequency  directed  by  the 
Commander  of  MOG  1  (CMOG  1)  as  determined  by  ships'  programmes,  which  will  dictate  both 
the  availability  of  ships  to  participate  and  the  level  of  participation  from  each  ship,  as  well  as  any 
special  training  required  to  help  prepare  for  upcoming  operations  or  exercises,  such  as  live  missile 
firings.  CMOG1  will  dictate  the  scope  and  intensity  of  the  exercise,  and  these  may  vary  from  basic 
operator  training  to  full  combat  team  training.  One  ship  will  be  designated  as  the  Officer 
Conducting  the  Serial  (OCS)  for  each  exercise  and  will  be  responsible  for  developing  an 
appropriate  scenario,  generating  the  required  instructions,  and  acting  as  the  control  staff  during  the 
running  of  the  scenario. 

In  order  to  participate,  ships  must  have  a  functioning  CCS,  Link-1 1  and  radio  suite.  Ships  will 
implement  a  communication  plan  just  as  they  would  at  sea.  Normally  UHF  communications  are  used. 
All  required  tactical  voice  and  Link-1 1  circuits  will  be  established,  as  well  as  a  separate  voice  circuit 
used  by  the  OCS  to  co-ordinate  the  serial  with  other  ships.  Within  each  ship  honest  brokers  known  as 
"gamemasters"  will  relay  or  input  information  as  directed  by  the  OCS  gamemaster.  These 
gamemasters  are  normally  combat  operators,  and  will  always  be  ship’s  own  crew. 

The  CCS  in  each  participating  ship  will  be  placed  in  "Sim",  which  facilitates  the  realistic 
simulation  of  weapons  systems  and  other  simulation  functions  such  as  the  generation  of  CCS 
symbology  once  a  radar  "detects"  a  simulated  contact.  For  this  feature  to  work  the  ship  must  have 
its  radar  antennas  rotating,  but  not  necessarily  emitting.  When  ships  "detect"  contacts  that  have 
been  input  by  the  gamemasters,  they  react  in  accordance  with  standard  operating  procedures.  This 
would  include  Linking  the  track  to  other  ships,  as  well  as  amplifying  the  track  according  to 
available  real  or  simulated  sensor  information,  scenario  intelligence  etc. 

Each  exercise  normally  takes  place  over  a  forenoon.  The  initial  set-up  will  normally  take  about  an 
hour  as  communications  and  Link  are  established,  and  background  information  is  entered  into  the 
CCS.  The  actual  scenario  might  run  for  1  Vi  -  2  Vi  hours  depending  on  the  aim  and  complexity  of 
the  exercise.  After  the  scenario  the  OCS  will  normally  conduct  a  debrief  over  a  radio  circuit  and 
invite  input  from  all  ships.  A  hard  copy  message  summarising  the  exercise  will  be  submitted  by 
participating  ships  to  CMOG1.  Ships  will  also  conduct  internal  debriefs.  It  is  not  common  practise 
to  use  ships'  internal  data  recording  capabilities  during  these  debrief  sessions,  but  they  could  be 
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used.  Ships  could  use  C-SAVRAS,  CCS  History  Record  or  any  other  data  capture  devises  (bus- 
sniffer)  that  could  be  used  by  ships  at  sea,  providing  these  are  fitted. 

Advantages 

•  Exercises  are  regularly  scheduled. 

•  Real  Ops  Rooms  are  used  so  there  is  a  high  degree  of  face  validity  in  terms  of 
equipment  capabilities  and  layout. 

•  Real  ships  teams  are  used,  including  technicians. 

•  Simulation  facilitates  realistic  weapons  engagements. 

•  Numerous  Ops  teams  can  participate  simultaneously. 

•  Data  can  be  captured  the  same  as  for  a  ship  at  sea. 

•  The  infrastructure  exists  for  these  exercises  to  be  conducted  on  a  regular  basis.  Ops 
teams  are  familiar  with  their  conduct. 

•  Supporting  players  available  to  play  the  necessary  background  roles  to  feed  data  to 
scenarios,  move  game  pieces  etc. 

•  Capability  to  insert  T&E  real-time  data  into  scenarios. 

•  Numerous  scenarios  of  varying  complexity  are  resident  in  various  ships  and  MOG  1. 

•  Available  tools,  expertise  and  knowledge  to  build  scenarios. 

Disadvantages 

•  T&E  would  need  to  receive  priority  to  dedicate  an  exercise  for  T&E. 

•  There  is  limited  capability  to  input  realistic  data  to  stimulate  acoustic  and  ESM 
sensors. 

•  Only  two  pieces  of  radar  video  can  be  presented  in  each  ship  (a  function  of  the  SPS  49 
air  search  radar).  Otherwise,  ships  are  fighting  symbology. 

•  The  expertise  of  various  ships*  teams  to  run  scenarios  and  use  the  Sim  function  of  the 
CCS  is  inconsistent. 

•  Additional  video  and  audio  data  would  have  to  be  manually  integrated  into  existing 
recording  capability. 

•  Reduced  data  may  not  be  taken  off-site  for  subsequent  analysis. 

•  No  recording  of  individual  face  to  face  conversations  without  additional  microphones. 

•  No  capability  to  integrate  MSDF  functions  into  existing  C2  software. 

•  The  normal  ships’  routine  carries  on,  including  superfluous  pipes  over  the  main 
broadcast. 
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3.5.  Ops  Room  of  Ship  during  sea  exercise 

Essentially  this  environment  would  have  the  similar  capability  and  limitations  for  T&E  purposes  as 
the  Wednesday  War  configuration.  This  similarity  exists  both  for  the  creation,  running  and  data 
collection  aspects  of  exercise  scenarios.  Some  of  the  major  differences  obviously  involve  the 
additional  elements  of  fatigue,  stress  and  motivation  that  may  be  absent  from  harbourside 
exercises.  This  can  be  seen  as  adding  to  the  validity  of  the  evaluation.  The  sea  exercise  also  has 
some  advantages  over  the  latter,  for  example: 

•  There  is  more  dedicated  technical  support  for  the  exercise. 

•  All  positions  are  manned. 

•  There  are  no  mundane  interruptions  that  have  nothing  to  do  with  the  exercise  of  the 
type  that  may  occur  when  the  ship  is  at  harbour. 

•  Real  radar  returns  are  present  (e.g.  if  conducted  in  proximity  to  an  airway,  commercial 
traffic  will  be  present  on  radar  screens). 

•  Because  the  ship  is  on  a  24/7  schedule,  there  is  a  potential  for  more  availability  for 
T&E  purposes  than  a  ship  alongside. 

Some  of  the  disadvantages  (over  the  harbourside  Wednesday  War)  include: 

•  The  only  available  technical  support  is  on  board  ship  (i.e.  unlike  at  harbour,  where 
additional  resources  may  be  brought  in). 

•  The  length  and  specific  scheduling  of  sea  trials  may  be  difficult  to  integrate  with  T&E 
planning. 

•  High  level  authority  would  be  required  to  permit  T&E  functions  and  associated 
personnel  to  be  aboard  during  exercises. 
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3.6.  Existing  Navy  C2  data  capture  and  analysis  systems 

In  this  section  we  describe  two  systems  (and  related  functionality)  that  have  been  developed  to 
capture  aspects  of  combat  system  data  within  the  existing  Ops  Room  infrastructure.  These  are  the 
Combat  System  Audio  Visual  Recording  and  Analysis  System  (C-SAVRAS)  and  the  System 
Analysis  Simulation  Evaluation  (SASIE). 

3.6.1.  C-SAVRAS 

This  system  has  two  main  capabilities:  (i)  video  capture  and  replay  of  the  content  of  all  CCS 
screens  in  the  Ops  Room,  and  (ii)  audio  capture  of  communications  on  networks.  Its  output  may  be 
integrated  with  data  captured  by  three  other  functions:  History  Recording,  Bus  Sniffer  and  SASIE. 

The  video  capture  capability  of  C-SAVRAS  essentially  provides  a  video  recording  of  the  content 
of  any  CCS  console  that  can  be  subsequently  played  back  upon  a  standard  television  monitor.  It 
should  be  noted  that  the  samples  of  playback  that  were  provided  showed  considerable  loss  of 
picture  quality  for  area,  text  and  symbology  data.  Further,  experienced  Navy  personnel  are 
required  to  make  their  best  estimate  and  interpret  the  displayed  data.  The  captured  video  data  can 
also  be  digitised  and  exported  as  MPEG  files  onto  a  CD.  A  maximum  number  of  eight  positions 
can  be  monitored,  but  this  may  be  further  reduced  by  the  number  of  available  video  recorders.  It 
takes  less  than  an  hour  to  set  up  the  recording  capability.  Parallel  recordings  from  separate 
workstations  are  not  integrated  into  a  common  timebase.  The  audio  capture  on  tape  is  achieved 
through  the  operators'  microphone  which  is  always  "hot"  plus  whatever  information  that  the 
operator  receives  through  the  headset.  In  playback  there  is  no  capability  to  identify  the  source  of  a 
message,  instead  it  must  be  inferred  from  the  context. 

History  record  provides  a  record  of  information  at  the  CCS  level  and  includes  messages,  the  state 
of  the  system,  information  displayed  (e.g.  track  data)  and  console  interactions,  (e.g.  QAB).  The 
resulting  data  can  be  printed  out  as  a  serial  list,  dumped  to  a  PC  or  converted  to  an  ACCESS 
database  using  a  parser.  The  original  captured  data  (i.e.  not  the  processed  data)  can  only  be 
replayed  on  a  CCS  and  there  is  some  limited  control  over  playback  speed.  History  Record  has  a 
limited  capacity,  which,  in  the  worst  case,  would  correspond  to  about  6-7  minutes  of  data  for  AAW 
functions.  An  underlying  log  clock  records  events  to  the  millisecond  level,  but  the  reliability  and 
accuracy  of  measurement  to  this  precision  is  unknown.  The  user  may  also  set  the  required  scale  of 
time  measurement. 

Logistical  support  is  provided  by  ship-based  resources  and  the  quality  of  the  data  obtained  is  highly 
dependent  upon  the  skill  and  experience  level  of  the  support  staff. 

The  Bus  Sniffer  is  a  data  monitoring  system  that  captures  data  from  the  serial  data  buses  that 
support  the  CCS  and  other  systems.  Using  the  HR  and  Bus  Sniffer  together  provides  the  capability 
of  generating  a  data  file  that  comprises  the  following  elements:  event  number,  event  time,  source, 
message  type  (e.g.  vehicle  track)  and  details  associated  with  the  specific  message  type.  An 
example  of  output  of  data  captured  by  the  bus  sniffer  is  given  below. 
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Figure  9:  Example  Data  Table  derived  from  the  Bus  Sniffer 
3.6.2.  SASIE 

The  Bus  Sniffer  and  HR  data  can  be  also  be  integrated  with  the  SASIE  software  developed  by 
Thomson-CSF.  This  provides  a  capability  to  capture  and  record  data  via  the  bus  sniffer  between  the 
CCS,  sensors.  Link  1 1  and  weapons  systems  on  HALIFAX  Class  ships.  SASIE  has  its  own  replay 
and  analysis  tools  and  the  data  can  be  downloaded  to  a  PC  in  the  format  of  a  sequential  data  items 
file.  This  file  can  also  be  transformed  and  replayed  graphically  to  show  data  and  events  plotted  in 
their  spatial  and  temporal  context.  For  example,  the  timeline  data  can  be  converted  to  a  polar  plot 
to  allow  geographical  reconstruction  of  the  ASW  track  history.  Link  1 1  data  are  recorded  in  audio 
format  then  converted  by  software  to  text  data  for  incorporation  into  the  SASIE  event  file.  Sonar 
data  must  be  captured  separately  on  magnetic  tape  and  can  only  be  replayed  off-site.  The  sonar 
replay  provides  the  acoustic  data  only,  not  the  actions  of  the  operator. 

In  general,  SASIE  is  used  to  reconstruct  events,  usually  associated  with  exercises  by  recreating  an 
engagement  timeline.  Data  items  not  captured  by  SASIE  must  be  manually  added.  In  general, 
most  analyses  of  operational  exercises  tend  to  be  qualitative  rather  than  quantitative  and  there  is  no 
capability  for  statistical  analysis,  since  the  "n"  is  low  and  variability  high,  due  to  the  large  number 
of  variables  that  are  typically  encountered  during  actual  exercise  conditions.  SASIE  is  constrained 
currently  by  tape  length  to  capture  a  maximum  of  1000  data  items  in  a  single  session.  Sample 
SASIE  data  items  are  shown  in  the  next  figure. 
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Figure  10:  Example  Data  Table  derived  from  the  SASIE 

In  addition  to  SASIE  there  are  two  other  relevant  analysis  tools. 

•  Timeline  (developed  by  MARLANT  Operations  Research)  which  provides  a  static 
representation  of  all  engagement  data  versus  time  that  is  platform  centric  and  show 
target  versus  ground  truth  in  range,  bearing,  altitude/depth  and  presents  sensor  and 
command  actions. 

•  MEXANS  -  Maritime  Exercise  Analysis  Suite  (DSTO-AUSTRALIA)  is  used  to 
reconstruct  and  analyse  events  that  occurred  during  a  maritime  patrol,  exercise  or  trial. 
This  tool  plots  ‘ground  truth’  data  against  data  recorded  on  board  a  combat  system. 


3.6.3.  Summary  of  Advantages  and  Disadvantages 

Advantages 

•  Ability  to  collect  actual  operational  data  under  actual  exercise  conditions. 

•  Captures  content  of  the  CCS. 

•  Captures  comm  traffic. 

•  Could  be  implemented  in  the  CSTC  for  data  capture. 

•  Captures  data  on  a  common  timebase. 

•  T&E  data  communication  data  could  be  input  in  real  time. 

•  Tools  are  available  for  some  aspects  of  data  analysis. 
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•  Data  can  be  downloaded  to  a  PC  for  analysis. 

Disadvantages 

•  Video  replay  data  may  lack  the  appropriate  resolution  to  identify  symbology  and 
alphanumeric  data  for  T&E  needs. 

•  Severe  limitations  on  length  of  data  records,  particularly  for  AAW. 

•  Requires  trained  personnel  to  set  up  and  operate.  Data  interpretation  and  reliability  is 
dependent  upon  skill  level  of  support  personnel. 

•  Availability  of  support  personnel  highly  dependent  upon  ship  scheduled  activities. 

•  Could  not  accommodate  the  input  of  T&E  data  associated  with  sensors  or  tracks.  That 
is,  the  only  way  a  T&E  track  could  be  introduced  would  be  through  a  Link  1 1  or  audio 
comm. 

•  Can  only  operate  in  the  CSTC  or  on  board  ship. 

3.6.4  Conclusions 

Overall,  the  above  systems  provide  a  capability  for  capturing  operational  data  from  Ops  Room 
sized  exercises  under  highly  realistic  conditions.  The  data  capture  and  analysis  capability  has 
largely  been  developed  to  analyse  missions  qualitatively  with  a  view  to  diagnosing  and  rectifying 
operational  procedures  and  assessing  the  performance  of  the  Ops  Room  team.  Experience  has  been 
accumulated  in  using  the  systems  and  an  infrastructure  of  support  personnel  to  create,  control  and 
analyse  exercises  is  in  place.  However,  since  the  support  personnel  for  control  and  data  capture  are 
part  of  a  ship's  company,  their  availability  to  provide  support  for  T&E  activities  may  be  even  less 
than  the  support  personnel  for  CSTC  or  NCOT. 

Given  the  overall  size  and  availability  of  the  ship-based  infrastructure,  it  seems  appropriate  to  only 
consider  using  C-SAVRAS  and  associated  software  in  the  context  of  at-sea  exercises  at  the 
individual  warfare  level.  The  level  of  infrastructure  could  not  be  justified  for  capturing  T&E  data 
for  single  team  members  doing  single  tasks,  sub-tasks  or  task  strings.  Further,  the  ability  of  the 
T&E  team  to  provide  real  time  control  over  scenario  events  is  constrained  by  both  operational 
circumstances  and  system  capabilities.  In  addition,  the  large,  uncontrolled  variability  associated 
with  collecting  data  during  operational  exercises  and  the  constraints  on  generating  suitable  sample 
sizes,  precludes  any  rigorous  statistical  analysis  of  the  data  obtained. 

SMEs  knowledgeable  in  SASIE  suggest  that  it  would  be  most  useful  for  analysing  the  detect-to- 
engage  process  at  the  air  and  surface  warfare  team  level,  and  we  concur  with  this  assessment.  The 
Wednesday  War  configuration  may  prove  useful  to  rehearse  sea  trial  scenarios  prior  to  their 
conduct  to  iron  out  procedural  issues.  Furthermore,  simple  air  detect-to-engage  scenarios  should  be 
cross-validated  between  simulation  environments  to  compare  results. 
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4.  Availability  of  personnel  as  test 
participants 


In  general,  since  all  of  the  facilities  under  consideration  are  under  Navy  jurisdiction  and  the 
expected  test  participants  will  be  Navy  personnel,  then  the  same  constraints  and  availability  will 
apply  to  all  environments.  Dedicated  access  to  such  Navy  personnel  for  the  Ops  Room  at  harbour 
or  at  sea  may  be  more  restrictive,  because  of  other  demands  on  such  personnel.  In  general. 
Command  authority  will  be  needed  to  ensure  that  the  COMDAT  evaluation  project  is  given  priority 
for  access  to  personnel.  Further  research  into  availability  will  be  necessary  once  the  specific 
requirements  for  test  participants  have  been  developed  later  in  the  project. 

It  remains  possible  that  if  some  of  the  functionality  of  NCOT  can  be  run  on  a  workstation  outside 
of  NCOT,  say  at  DCIEM,  DREA  or  a  contractor's  site,  then  non-Navy  personnel  may  be  able  to  act 
as  test  participants.  This  activity  would  be  limited  to  assessments  of  simple  task  functions  only, 
where  operational  experience  would  not  be  necessary  and  adequate  training  could  be  provided. 

Given  the  long  term  nature  of  the  COMDAT  program  and  the  obvious  advantages  of  a  properly 
conducted  simulator  T&E  environment  systematically  linked  to  sea  trial  data,  it  is  suggested  that 
there  be  a  regular  scheduled  annual  commitment  of  simulator  time  and  support  for  T&E  purposes. 


Page  42 


Review  of  T&E  Environments  for  COMDAT  MOPs 


Humansyj/emj  Incorporated 


P516804.PDF  [Page:  54  of  100] 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


Humanyystems  Incorporated 


Review  of  T&E  Environments  for  COMDAT  MOPs 


Page  43 


P516804.PDF  [Page:  55  of  100] 


5.  Overall  Conclusions 


5.1 .  Summary  and  of  findings 

In  order  to  provide  a  summary  assessment  and  to  allow  comparisons  across  the  potential  test 
environments,  a  broad  assessment  was  made  of  each  environment  against  the  core  T&E 
requirements  listed  in  section  2.6.3.  This  assessment  was  performed  using  the  following  three 
categories: 

■  the  facility  satisfies  a  particular  requirement  (indicated  by  a  check  mark  below) 

■  the  facility  may  or  may  not  meet  a  requirement  (indicated  by  a  question  mark) 

■  the  facility  does  not  meet  the  requirement  (indicated  by  an  "X"  below). 

The  overall  assessment  was  based  upon  a  review  by  two  members  of  the  HSI®  team  of  the  detailed 
descriptive  data  and  comments,  and  then  a  subjective  assessment  of  which  category  to  assign.  In 
performing  this  task,  certain  of  the  descriptor  items  were  invariably  deemed  to  be  more  important 
than  others,  although  no  formal  weights  were  assigned,  in  affecting  the  overall  judgment.  Once  the 
overall  assessment  had  been  performed  it  was  reviewed  by  the  remaining  members  of  the  HSI® 
team  and  significant  consensus  was  obtained.  Some  discussion  emerged  on  a  few  items  for  some 
of  the  test  environments  that  resulted  in  their  evaluation  category  being  changed. 


The  following  table  provides  the  result  of  this  process  and  represents  a  high  level  summary  of  this 
assessment  of  the  ability  of  each  of  the  potential  test  environments  to  meet  the  needs  of  T&E. 
Numbers  in  a  cell  refer  to  subsequent  explanatory  notes. 


ORTT 

NCOT 

CSTC 

Ops  Room- 
Harbourside 

Ops  Room- 
Ship  at  Sea 

Availability  of  the  facility 

?1 

?1 

?1 

?2 

X 

Appropriate  Workstations  (to 
the  specific  T&E  paradigm) 

✓ 

✓ 

✓ 

Flexible  arrangement  of 
workstations  for  T&E 

✓ 

X 

X 

X 

Capability  to  simulate  the 
functions  required  to  support 
the  team  role  or  roles  under 
analysis 

✓ 

?3 

✓ 

✓ 

✓ 

Capability  to  create  T&E 
scenarios 

✓ 

✓ 

✓ 

?4 

?5 

T&E  personnel  can  create 
scenarios  more  or  less 
autonomously 

X 

*  6 

X 

X 

X 

Capability  to  control  T&E 
scenarios  in  real  time 

✓ 

✓ 

?  7 

?  7 

Capability  to  capture  data  to 
the  required  level  of 
precision 

✓ 

?8 

?8 

?8 

Capability  to  capture  data 
reliably 

* 

✓ 

?8 

?8 

?8 
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Capability  to  capture  the 
required  volume  of  data 

* 

* 

X 

X 

X 

Capability  for  reducing  and 
analysing  the  data 

* 

✓ 

?9 

?9 

?9 

Capability  to  run  repeated 
trials  for  limited  Ops  Room 
positions 

*  10 

*  10 

X 

X 

Explanatory  Notes: 

1.  These  facilities  are  in  high  demand  for  training  purposes.  If  time  is  to  be  allocated  for  T&E 
purposes  then  it  will  need  to  be  booked  in  April  for  that  particular  calendar  year.  In  the  present 
year,  there  may  be  as  few  as  3-4  days  available  in  the  ORTT. 

2.  Access  to  this  environment  may  be  even  more  difficult  since  it  depends  on  the  availability  of  at 
least  two  ship's  support  personnel,  as  well  as  the  constraints  imposed  by  ongoing  training 
exercises. 

3.  As  yet,  NCOT  remains  unproven  in  its  ability  to  simulate  functions  performed  at  the  warfare 
team  level  (and  higher)  that  require  multiple,  interacting  Ops  Room  personnel  supported  by  the 
necessary  game  controllers  and  players. 

4.  Scenario  creation  would  be  dependent  upon  the  development  of  scenarios  initially  for  the 
CSTC,  since  there  are  significant  constraints  on  the  availability  of  shipboard  personnel  to  assist 
this  process. 

5.  Scenarios  at  sea  appear  to  be  limited  to  specific  Navy  training  requirements.  At  present,  these 
tend  to  be  narrow  in  scope  (e.g.  weapons  and  other  responses  to  live  targets)  and  do  not  meet 
the  required  range  of  scenarios  that  would  be  needed  for  T&E. 

6.  Would  require  suitable  training  by  MD 

7.  Such  exercises  (especially  at  sea)  are  subject  to  a  number  of  uncontrollable  variables  that 
prevent  even  a  well-planned  scenario  from  being  executed  without  variance  over  repetitions. 

8.  The  existing  data  capture  tools  are  not  designed  to  meet  the  more  demanding  requirements  of 
T&E.  Video  data  are  of  low  resolution  and  require  subjective  interpretation  by  experienced 
personnel.  The  general  ability  to  capture  reliable  data  is  highly  contingent  upon  the  skill  level 
and  experience  of  game  controllers  and  analysers. 

9.  This  capability  is  limited  to  existing  functionality  designed  for  Navy  needs.  It  would  need  to 
be  expanded  and  further  developed  to  meet  the  requirements  of  T&E. 

10.  Limited  ability  to  re-configure  workstations  for  different  roles. 

5.2.  Fidelity  Issues 

As  we  have  noted  in  Matthews  et  al  (1997),  the  fidelity  requirements  of  a  test  bed  can  range  from 

the  simple  to  the  complex.  Four  levels  of  test  bed  configuration  were  identified. 

•  Level  1 :  low  fidelity  representation  of  console,  screens,  or  floor  layout, 

•  Level  2:  static  test  screens  representing  basic  system  part  task  functions. 
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•  Level  3:  part  system  functionality  with  task  strings  performed  interactively  between 
positions, 

•  Level  4:  fully  functional  prototypes  configured  for  typical  team  operation. 

Based  on  the  above,  NCOT  appears  to  have  a  fidelity  at  least  at  level  3  and  the  ORTT,  CSTC  and 
Ops  Rooms  alongside  or  at  sea  at  level  4. 

In  general,  it  appears  possible  to  rank  order  the  fidelity  of  the  four  test  environments  from  lowest  to 
highest  as  follows:  NCOT<  ORTT=CSTC<Ops  Room  alongside<  Ops  Room  at  sea.  Of  course,  the 
latter  two  rankings  are  hardly  surprising. 

NCOT  has  lower  fidelity  mostly  in  terms  of  the  physical  aspects  of  the  facility,  that  is  the  hardware 
itself,  the  look  and  feel  of  the  hardware,  the  layout  of  the  workstations  and  ancillary  equipment  and 
the  mode  of  communications.  However,  from  a  functional  perspective  the  system  appears  capable 
of  being  able  to  support  assessment  of  complex  task  strings  involving  teams.  The  major  increase  in 
fidelity  that  is  found  in  the  ORTT  and  CSTC  is  due  to  its  very  close  similarity  to  the  physical 
environment  of  a  HALIFAX  Class  Ops  Room.  A  task  environment  that  involves  the  whole  Ops 
Room  personnel  interacting,  moving  and  communicating  with  each  other,  as  they  would  at  sea,  can 
be  reproduced  with  a  high  degree  of  fidelity  in  the  ORTT  and  CSTC,  but  that  does  not  appear 
possible  for  NCOT,  or  to  a  much  lesser  degree. 

5.3.  Analysis  of  findings 

Based  upon  what  we  have  learned  to  date,  we  have  made  a  preliminary  assessment  about  the  most 
appropriate  environment  for  performing  the  various  types  of  T&E  Functions,  as  indicated  in  the 
table  below. 

This  assessment  is  based  on  the  general  approach  recommended  in  Matthews  et  al  (1997)  in  which 
the  test  environment  is  matched  to  the  testing  needs.  Further,  there  is  an  effort  to  avoid  wasting 
valuable  resources  by  proposing  an  environment  that  has  a  higher  fidelity  or  level  of  data  collection 
sophistication  than  a  specific  T&E  paradigm  might  require  (e.g.  using  the  ORTT  for  sub-tasks). 


Source 

Task 

performance 
(timing  and 
accuracy) 

Communication 

measures 

Situation 

awareness 

Decision 

Making 

Baseiine  Data 

Single  player 

Sub-task 

NCOT 

NCOT 

NCOT 

NCOT 

Task 

NCOT 

NCOT 

NCOT 

NCOT 

NCOT 

NCOT 

NCOT 

NCOT 

Function 

NCOT 

NCOT 

NCOT 

NCOT 

Warfare  team 

Sub-task 

NCOT 

NCOT 

NCOT 

NCOT 

Task 

NCOT 

NCOT 

NCOT 

NCOT 

Task  string 

NCOT 

NCOT 

NCOT 

Function 

ORTT 

ORTT 

ORTT 
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Full  Ops  Room 

Sub-task 

ORTT 

ORTT 

ORTT 

ORTT 

Task 

ORTT 

ORTT 

ORTT 

ORTT 

CSTC/C-SAVRAS 

Task  string 

ORTT 

ORTT 

ORTT 

ORTT 

CSTC/C-SAVRAS 

Function 

ORTT 

ORTT 

ORTT 

ORTT 

CSTC/C-SAVRAS 

As  can  be  seen  from  the  above  table,  it  is  recommended  that  the  T&E  effort  be  concentrated  on  two 
facilities.  NCOT  for  paradigms  up  to  the  level  of  task  strings  performed  by  multiple  players  and 
the  ORTT  for  everything  more  complex.  It  is  possible  that  the  functionality  of  NCOT  may  also 
accommodate  complete  operational  functions  at  the  Warfare  team  level.  However,  rigorous  proof 
of  concept  will  be  required  to  determine  whether  or  not  NCOT  can  actually  meet  the  T&E 
requirements  at  the  level  of  team  functions. 

At  this  time,  the  uses  of  any  of  the  potential  environments  that  involve  the  existing  Ops  Room 
equipment,  (that  is  the  CSTC  and  harbourside  or  at  sea  configurations)  is  not  recommended.  There 
are  several  reasons  for  this.  First,  there  are  concerned  about  the  level  of  precision,  quality  and 
quantity  of  data  that  can  be  captured  for  T&E  purposes.  For  example,  the  video  playback 
capability  of  C-SAVRAS  does  not  seem  adequate  for  assessing  the  detailed  content  of  the  CCS. 
Also  we  have  been  told  that  the  quality  of  the  data  is  highly  dependent  upon  the  skill  level  of  the 
support  personnel.  The  volume  of  data  that  can  be  collected  is  constrained  by  existing  limits  of  the 
recording  systems.  Second,  it  may  be  difficult  to  integrate  new  functions  at  the  software  and 
hardware  level  associated  with  MSDF  into  the  existing  system.  (Just  as  it  has  been  difficult  to 
integrate  data  capture  methods,  resulting  in  less  than  optimum  solutions  being  implemented,  such 
as  the  bus  sniffer).  Third,  availability  of  the  harbourside  and  at-sea  configurations  is  likely  to  be 
highly  constrained  and  unlikely  to  meet  the  somewhat  extensive  testing  requirements  for  evaluating 
MSDF.  Fourth,  it  appears  that  data  collection  at  sea  is  at  the  mercy  of  a  variety  of  uncontrolled 
variables  that  may  preclude  the  collection  of  reliable,  repeatable  and  stable  performance  data. 

The  CSTC/C-SAVRAS  combination  may  be  useful  for  providing  some  baseline  data  at  the  gross 
function  level,  for  example  the  time  it  takes  for  the  detect-to-resolve  cycle,  and  for  assessing 
communication  patterns  and  loads  in  a  variety  of  operational  tasks.  Further,  there  may  be  some 
useful  data  in  existing  exercise  databases  captured  in  the  facility  or  at  sea  that  may  provide  some 
broad  basis  for  comparing  the  impact  of  MSDF. 

With  respect  to  NCOT,  there  is  a  reservation  at  this  time  in  considering  it  as  a  primary  candidate 
for  T&E  needs  in  terms  of  its  ability  to  actually  deliver  the  required  functionality.  It  should  be 
remembered  that  much  of  what  we  have  learned  is  by  word  of  mouth  from  the  MD  company 
representative  and  the  documentation  that  describes  the  proposed  functionality.  There  is  also 
currently  very  little  accumulated  experience  in  using  the  facility  for  anything  other  than  computer 
based  training.  Thus,  there  is  a  very  large  question  at  present  concerning  the  NCOT's  proven 
ability  to  allow  development  of  the  required  T&E  software,  to  meet  requirements  for  the  set  up  and 
control  of  data  collection  sessions,  and  to  reliably  collect  data  that  are  in  a  format  amenable  to 
analysis.  Nevertheless,  based  upon  the  documentation  on  system  capabilities,  we  remain  hopeful 
that  as  NCOT  is  developed  that  it  will  realise  its  potential  for  T&E  activities  over  the  lengthy 
timeframe  of  the  COMDAT  program. 

The  ORTT  appears  to  provide  all  of  the  basic  requirements  for  T&E  purposes  for  complex 
operational  functions  and  would  be  the  clear  first  choice  for  running  team  scenarios  that  involve 
functions  at  the  warfare  team  level  (e.g.  detect-to-  resolve)  and  for  all  scenarios  involving  the 
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complete  Ops  Room  team.  It  is  not  considered  a  better  alternative  for  T&E  at  the  level  of 
individual  tasks  and  task  strings  for  several  reasons.  First  the  level  of  overhead  may  be 
unwarranted  for  such  relatively  straightforward  T&E  paradigms.  Second,  it  does  not  lend  itself 
quite  as  well  as  NCOT  to  flexible  configuration  for  multiple  subjects  to  be  tested  in  parallel  doing 
the  same  task  on  multiple  workstations. 

Finally,  there  are  some  major  concerns  about  being  able  to  access  the  various  facilities  and  in 
obtaining  the  level  of  support  required  to  run  T&E  scenarios.  This  concern  stems  from  the  high 
demand  from  the  Navy  to  meet  their  training  requirements,  the  continuing  use  of  NCOT  and  the 
ORTT  by  the  commercial  developers  and  the  constraints  on  availability  of  support  personnel.  The 
most  optimistic  assessment  is  that  the  ORTT  could  be  made  available  for  3  days  in  the  current 
calendar  year,  if  the  reservation  were  made  before  the  end  of  April.  NCOT  may  be  more  available, 
but  again  early  booking  would  be  required. 
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6.  Existing  MOPs 


Quantitative  MOPs  that  would  provide  baseline  measures  for  T&E  purposes  do  not  appear  to  exist. 
Some  quantitative  MOPs  concerning  operational  functions  such  as  weapons  responses  to  incoming 
threats  have  been  developed  to  collect  selected  data  during  live  exercises  at  sea.  Such  data  are 
probably  of  limited  value  for  baseline  purposes.  First,  they  tend  to  be  highly  sensitive  and  hence 
access  may  be  a  problem.  Second,  the  sample  size  is  very  small.  Third,  the  reliability  of  the  data 
may  be  compromised  by  a  variety  of  uncontrolled  variables  that  tend  to  occur  with  live  exercises. 

For  the  ORTT,  and  certainly  for  NCOT,  there  does  not  appear  to  be  as  yet  an  accumulated  body  of 
quantitative  data  that  could  be  readily  accepted  as  representing  baseline  performance.  We  have  yet 
to  see  samples  of  data  sets  that  have  been  collected  in  the  CSTC  during  simulation  exercises. 
However,  given  the  purposes  of  such  data  collection  (training  feedback),  concerns  we  have 
expressed  earlier  over  the  level  of  precision  required  for  T&E  purposes  and  the  fact  that  we  would 
be  trying  to  fit  the  data  after  the  fact  to  the  T&E  paradigm,  suggests  that  even  if  such  data  were 
available  in  quantity  its  usefulness  would  be  limited. 

Hence,  the  thrust  of  the  future  work  to  develop  MOPs  will  guided  by  the  general  performance  areas 
that  seem  to  be  of  operational  interest,  rather  than  the  adoption  of  existing  metrics.  This 
development  process  will  also  be  guided  by  previous  work  by  Humansystems  and  DCIEM  in  this 
field,  work  underway  at  DREV  and  the  available  literature  on  measuring  performance  in  C2. 
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7.  Next  Steps 


Our  preliminary  recommendation  is  to  set  up  a  limited  proof  of  concept  trial  for  NCOT.  This 
would  involve  the  following  steps: 

•  Assemble  a  small  T&E  team.  Walk  through  selected  individual  and  small  team 
existing  scenarios  with  trained  operators  and  trainers  for  familiarisation  purposes. 

•  Work  with  Navy,  and  possibly  MD,  personnel  to  develop  (or  select  from  existing) 
scenarios  and  scenario  fragments. 

•  Develop  a  small  cross  section  of  MOPs  for  these  scenarios. 

•  Proof  the  scenarios  in  NCOT  and  collect  sample  data. 

•  Assess  ability  to  run  and  collect  data  on  single  players/single  tasks,  two  interacting 
players  on  single  tasks  and  multiple  players  on  the  same  single  task. 

•  Provide  MD  with  specifications  for  reduction  of  data  from  sample  data  files. 

•  Analyse  sample  data. 

We  also  strongly  recommend  that  DCIEM  initiate  the  process  of  reserving  space  with  the  necessary 
naval  authorities  for  carrying  out  the  above  steps  in  NCOT  in  the  Fall  of  2001  and  also  make 
reservations  for  2002  in  both  NCOT  and  the  ORTT.  Our  preliminary  suggestion  is  that  testing  will 
require  a  full  working  week  (five  days)  in  each  facility. 

Finally,  whether  dealing  with  NCOT  or  the  CSTC,  funds  will  be  need  to  be  found  to  support  the 
commercial  development  of  T&E  software  by  MD  and  LM.  The  specific  requirement  for  this  will 
be  detailed  in  the  final  deliverable  for  this  project. 


Page  52 


Review  of  T&E  Environments  for  COMDAT  MOPs 


Humans y stems  Incorporated 


P516804.PDF  [Page:  64  of  100] 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


Humamystems  Incorporated 


Review  of  T&E  Environments  for  COMDAT  MOPs 


Page  53 


P516804.PDF  [Page:  65  of  100] 


8.  References 


Canadian  Patrol  Frigate  ( CPF)  Operations  Room  Team  Trainer  ( ORTT)  System  Operator's 
Manual  (SOM).  Document  #202000516,  November  27,  2000  (CD  version). 

MacDonald,  Dettwiler  &  Associates  Ltd.  (1999)  NCOT  System  User’s  Manual,  Volume  1. 
Technical  System  Overview. 

MacDonald,  Dettwiler  &  Associates  Ltd.  (2000)  NCOT  System  Demonstration  CD. 

Matthews,  M.L.,  Webb  R.D.G  and  McCann,  C.  A  Framework  for  Evaluation  of  Military  Command 
and  Control  Systems.  DCIEM  Report  #.  May,  1997. 

Matthews,  M.L.,  Webb,  R.D.G.  and  Bryant,  D.  Cognitive  Task  Analysis  of  the  Halifax  Class 
Operations  Room  Officer.  DCIEM  Report  #  .  March  1999. 

Matthews,  M.L.  and  Webb,  R.D.G.  Validation  of  Cognitive  Task  Analysis  for  the  Halifax  Class 
ORO.  DCIEM  Report  #  .  March  2000. 

Powerpoint  Presentation  (and  CD)  Data  Collection  &  Operational  Analysis.  LCdr  K.M.  Brown, 
Canadian  Forces  Maritime  Warfare  Centre,  21  February  2001. 


Page  54 


Review  of  T&E  Environments  for  COMDAT  MOPs 


Human.vv.s/'t'ffi.v  Incorporated 


P516804.PDF  [Page:  66  of  100] 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


HumanYystejns  Incorporated 


Review  of  T&E  Environments  for  COMDAT  MOPs 


Page  55 


Annex  A:  Summary  Tables  of  Evaluation  Methods  and  Measures 


Table  Al:  Potential  Evaluation  Methods  and  Measures  (Function  Usability) 

(*  estimated  %  of  domain  subsection  evaluated  usng  this  configuration ) 
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•  For  each  task  relevant  variable 
users  can  promptly,  accurately  and 
completely 

-  Acquire  timely  information 

Consult  appropnate  sources. 
Detect  relevant  cues 

-  Comprehend  information 

-  Predict  immediate  future 

-  Maintain  information 

Store  (remember) 

Recall  (retneve) 

• 

(5% 

Questionnaire  data  •  Questionnaire  data' 

5  point  rating  scales  5  point  rating  scales 

completed  by  SME  and  HF  specialists  completed  by  tnal  participants 

Mean  estimates  given  by  SME  and  HF  Mean  ratings  given  by  tnal  participants 

specialists  for  “ Cntena ’  for  critical  tasks  for  each  position 

of  probability  of  typical  users  being  able  to  •  Performance  data 

fulfill  “ Cntena *  for  cntical  tasks  for  each  job  “Probe”  questions  to  determine  speed  and  accuracy  of  information 

position  acquisition 

evaluation)  Observed  response  to  new  information 

Observed  source  consultation 
*  %  checklist  information  sources  consulted 

•  Time  to  acquire  relevant  information 

-  Comprehension  of  information  acquired 

-  %  participants  responding  appropnately  to  info 

-  Response  time  to  information  status  changes 

-  SME  ratings  of  appropnateness  of  response 

-  Time  to  retneve  information  (etc)  (60%) 

•  Questionnaire  data 

5  point  rating  scales 
completed  by  tnal  participants 

Mean  ratings  given  by  tnal  participants 

for  “ Cntena ’  for  cntical  tasks  for  each  position 

•  Performance  data  (concurrent  tasks) 

“Probe”  questions  to  determine  speed  and  accuracy 
information  acquisition 

Observed  response  to  new  information 

Observed  source  consultation 
-  %  checklist  information  sources  consulted 

-  Time  to  acquire  relevant  information 

-  Comprehension  of  information  acquired 

-  %  participants  responding  appropnately  to  info 
-Response  time  to  information  status  changes 

-  SME  ratings  of  appropnateness  of  response 

-  Time  to  retneve  information  (etc  (35%) 

|1| 

•  For  each  task  and  message  type, 
users  can  minimize 
-  Communication  time  for 

Preparation 

Transmission 

Receipt 

Response 

-  Requests  for  clanfication 

-  Repeated  messages 

-  Errors  in 

Transmission 

Content 

• 

Questionnaire  data.  • 

5  point  rating  scales 

completed  by  SME  and  HF  specialists 

Mean  ratings  given  by  SME  and  HF  specialists 
for  “ Cntena ”  for  cntical  tasks  for  each  job 
position  • 

f  5%  evaluation) 

Questionnaire  data 

5  point  rating  scales 
completed  by  tnal  participants 

Mean  ratings  given  by  SME  and  HF  specialists 
for u Cntena ”  for  cntical  tasks  for  each  job  position 

Performance  data  (isolated  tasks) 

-  %  time  spent  on  communication 

-  Mean  transmissions/system  function 

-  Clanfications/1 00  messages 

-  Repeats/1 00  messages 

-  Mean  time  end  of  inquiry  to  start  of  response 

-  Transmission  errors/100  messages 

-  Content  errors/100  messages (*  20%  evaluation) 

•  Questionnaire  data 

5  point  rating  scales 
completed  by  tnal  participants. 

Mean  ratings  given  by  SME  and  HF  specialists 
for  “ Cntena ’  for  critical  tasks  for  each  job  position 

•  Performance  data'  (concurrent  tasks) 

-  %  time  spent  on  communication 

•  Mean  transmissions/system  function 

-  Clanfications/100  messages 
-Repeats/100  messages 

-  Mean  time  end  of  inquiry  to  start  of  response 

-  Transmission  errors/ 100  messages 

-  Content  errors/100  messages *  75%  evaluation) 

•  .<•* 

•  For  all  task  combinations  and 

• 

Questionnaire  data.  • 

Questionnaire  data 

*  Questionnaire  data. 

;  fi  j  v|  1 1 

durations,  the  system  permits  users  to 

5  point  rating  scales 

5  point  rating  scales 

5  point  rating  scales  completed  by  tnal  participants 

manage  mental  workload  to 

completed  by  SME  and  HF  specialists 

completed  by  tnal  participants 

SME  ratings  on-line  or  post  event. 

f  :  ' 

-  Maintain  effective  performance 
-Avoid  over-load 

Mean  workload  ratings  given  by  SME  and  HF 
specialists  for  “Cntena”  for  critical  tasks  for 

SME  ratings  on-line  or  post  event 

Mean  workload  ratings  by  participants  and  SMEs  (on-line  or  post- 

Mean  workload  ratings  by  participants  and  SMEs  (on-line 
or  post-scenario )  for  ‘ Crrtena  ’  for  concurrent  tasks  for 

1  . 

-Avoid  under-load 

each  job  position. 

C  devaluation) 

scenario)  for  “Cntena"  for  isolated  tasks. 

C  35%  evaluation) 

each  /Ob  position  and  different  traffic  volumes 
r  60%  evaluation) 

” ■, :*•  ij 

•  Relevant  layout  features  for  each  task 

• 

Guide  line  check  list  • 

Questionnaire  data- 

•  Questionnaire  data 

MJ  yjStl  ■’ {*  f  / 

[console  (person),  cell(team)  &  centre 

Anthropometnc  overlays 

5  point  ratings  of  discomfort  and  acceptability 

5  point  ratings  of  discomfort  and  acceptability 

sV  ; 

(crew)]  fit  users  physical  needs  for  • 

bnk  analysis 

completed  by  tnal  participants 

completed  by  tnal  participants 

*  Communication  links 

5  point  rating  scales 

Observer  ratings  of  video 

Observer  ratings  of  video 

* 

Visual  (line  of  sight,  eye  movt) 

completed  by  SME  and  HF  specialists 

Fitting  tnals  5-95  percentile  users 

Fitting  tnals  5-95  percentile  users 

1  j 

Auditory 

-  Estimated  workload  ratings  by  SME  and  HF 

-  Mean  acceptance  and  discomfort  ratings  by  participants  and  SMEs 

-  Mean  acceptance  and  discomfort  ratings  by 

-  Movement  links 

specialists  for  “Cntena”  for  each  job  position. 

for  Cntena”  for  isolated  tasks 

participants  and  SMEs  for  “Cntena ’  for  concurrent  tasks 

|gj  j  ~n  i4  |  ^  If 

Hand-arm,  Whole  body 

-  Predicted  %  of  users  able  to  complete 

-  Ability  of  OF  and  99*  percentile  users  to  complete  critical  links. 

-  Ability  of  9*  and  99h  percentile  users  to  complete  critical 

*t  •••  • 

-  Minimize  musculo-skeletal  stress. 

critical  links  (*30%  evaluation) 

C  40%  evaluation) 

links.  ("30%  evaluation) 
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see  Table  A 1 1  above  “Potential  Evaluation  Methods  and  Measures  User  Fir 


High  level  (unctions  in  operational  context 

•  Recorded  data  plus  SME  observer  and  tnai 
participant  ratings  of  acceptability  of' 

-  Situation  Awareness  (see  other  sections) 

-  Time  statistics  to  identify  objective(s) 

-  %  errors  in  determining  objective^) 

-  Time  statistics  on  generation  of  options 

-  Time  statistics  on  evaluation  of  options 

-  Time  statistics  on  presentation  of  options 

-  Suitability  of  options  chosen 

•  Impact  on  cntical  path  for  other  functions 
(65%  evaluation) 


For  given  mission  type,  users  can: 

*  Access  required  information 

-  Review  information  in  context 

-  Develop  /  maintain  awareness  of  situation 

-  Identify  objective^) 

-  Match  resources  to  objective 

-  Generate  alternative  options 

-  Evaluate  and  prioritize  alternatives 

-  Present  alternatives  to  others 

*  Decide  COA. 


•  HF  and  SME  estimates  and  ratings  of  acceptability  for 
•  Time  statistics  for  information  access 

-  %  completeness  of  information 

-  Situation  Awareness  (see  other  sections) 

-  Impact  on  cntical  path  for  other  functions 
(5%  evaluation) 


•  High  level  functions  performed  as  senes  of  task  stnngs 
Recorded  data  plus  SME  observer  and  tnal  participant 
ratings  of  acceptability  of. 

-  Situation  Awareness  (see  other  sections) 

-  Time  statistics  to  identify  objective^} 

-  %  errors  in  determining  objective^) 

*  Time  statistics  on  generation  of  options 

*  Time  statistics  on  evaluation  of  options 

*  Time  statistics  on  presentation  of  options 

-  Suitability  of  options  chosen 

-  Impact  on  cntical  path  for  other  functions 
(30%  evaluation) 
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Annex  B:  List  of  T&E  Requirements 


The  first  column  expands  upon  the  general  requirements  outlined  in  the  body  of  the  text  and  the 
second  column  contains  comments  and  issues  concerning  the  requirement. 


1 .6.  Availability  of  Technical/Controller  support  for 


1 .8.  Availability  of  Personnel  as  test  participants 


1 ,9.  Video  recording  available 


1,10.  Audio  recording  available 
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1.10.1.  Type  of  audio 


#  workstations  covered  (internal  vs  external  to  OR), 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources, 
identification  of  position(s),  incoming  vs  outgoing 
audio  quality  (noise,  etc) 
off  net  speech  in  OR 


1.9.1.  Type  of  video 


1 .9.2.  Eye  tracking  capability 


#  workstations  covered, 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources. 
Wide  area  coverage 

Macro  coverage  /  resolution  of  CCS  screen  data 
Quad  coverage 
Tracking  (pan,  zoom,  etc) 

Low  light  /  red  light _ 

Video, 

other 


1.8.1.  Number _ 

1 .8.2.  Experience  /  qualifications _ 

1 .8.3.  Individuals  or  teams _ 

1.8.4.  Notice  required _ 

1 .8.5.  Range  of  Ops  Room  team  represented 


1 .7.  Software/hardware  during  preparation 

1.7.1.  Software/hardware  during  testing 

1 .7.2.  Control  scenario  play 

1 .7.3.  Software/hardware  during  analysis 

1 .7.4.  Trg  of  HSl®  test  personnel  in  technical  issues 


Help  with  set  up  of  equipment 
Scenano  development  and  set  up 

Set  data  capture  equipment,  software,  analysis _ 

Maintenance  /  repair  of  crashes 

Ad  hoc  changes  to  scenano  (Unplanned  insertion  of  data) 

Availability  of  controller  staff  as  scenano  players,  war  game 

directors? _ 

Data  capture  and  analysis,  downloading,  configuring  for  other 

analysis  software,  etc. _ 

How  are  HSI®  staff  to  be  trained _ 

How  many  can  we  get? _ 

What  range  of  experience  and  qualification  (e.g.  students  only?) 

Will  they  have  worked  together  before? _ 

How  far  ahead  is  booking  required? _ 

Different  availability  for  different  levels?  Are  “filler”  personnel 
available  for  part  task  simulation? 
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1.11.  Information  exchange  (is  it  represented?) 

1.11.1.  Internal  net  in  OR 

Who  can  be  on  it  in  trainer? 

1 .1 1 .2.  External  nets  to  Ops  Room  on  ship 

Who  can  be  on  it  trainer? 

1.11.3.  External  nets  to  ship 

Who  can  be  on  it  trainer? 

1.11.4.  Text 

What  types  of  text  messages? 

Level  of  text  message  activity? 

1.11.5.  Face  to  face  in  OR 

Direct  voice  contact  in  OR? 

1 .1 1 .6.  Other  information  exchanges 

State  boards 

Manuals  /  Charts  /  Tacpacs  /  Post  its  /  etc. 

1.12.  Customise  environment  for  data  collection 

Ability  to  add  COMDAT  prototype  equipment  /  software 

Technical  support  for  this? 

1.12.1.  Extra  audio  and  video 

Add  extra  audio  /  video  data  capture  devices 

1 .1 2.2.  Add  on  software  modules 

Ability  to  add  on  software  modules  to  intenect  data  into  canned 
scenanos/simulations 

1.13.  Standalone  keypads,  notebooks  or  PDA's  for  data 
entry  by  participants  or  observers? 

Eg  for  workload  ratings  at  intervals 

Or  for  response  to  freeze  probes 
or  for  diary  entry 

1 .14.  Reliability  of  system 

Types  of  reliability  problems 

Crash  risk 

Repair  time  /  typical  down  time 

Repeatability  of  scenarios 

1.15.  Space  for  observers 

Is  there  space  for  more  people  to  observe  /  capture  data. 

Worst  case  =  one  observer  at  each  work  station. 

■5* 

2.1 .  Environmental  control 

(lighting,  acoustics,  thermal,  motion) 

Adjustability  and  control  over  lighting 

Adjustability  and  control  over  acoustics 

2.2.  Pre-packaged  scenario 

Availability? 

Modifiabilty?  (e.g.  freeze  and  tracking  probes) 

2.3.  Complexity  of  scenarios 

2.3.1.  Mission  types 

Location,  threat  type,  TG  configuration,  OOTW,  Aid  civil  power,  etc 

2.3.2.  Mission  segments 

Part  task  breakdown  capability 

2.3.3.  Number  of  players 

Multiple  players  on  single  task,  or  single  player  multiple  roles. 

2.3.4.  Number  and  types  of  assets 

How  many  and  what  types  en  /  fr  /  neutral  ships,  a/c,  weapons, 
sensors,  etc. 

Can  these  be  vaned? 

2.3.5.  Number  and  levels  of  team  roles 

Can  these  be  varied? 

2  4.  Ability  to  run  pre-packaged  scenano  to  a  defined 
timeline 

Certain  events  at  certain  points  on  time  line,  regardless  of  player 
response. 

Data  to  be  collected  on  that  time  line. 

2.5.  Ability  to  control  select  variables  in  real  time 

Ad  hoc  failures  of  Circuit  or  Net,  workstation  to  fail,  weapon 
system  to  fail. 

2.6.  Ability  to  interject  probes  in  real  time 

Freeze  probes 

Tracking  probes 

2.7.  Ability  to  stop/re-start  scenarios 

2.8.  Ability  to  backup/re-run  scenarios 

2.9.  Ability  to  dynamically  modify  scenario  depending  upon 
responses 

If  x  then  y  e.g.  if  fails  to  respond  to  threat,  re-insert  same  threat 
later 

If  a  then  b 

2.1 0.  Comms  among  experimenters  /  controllers 

Can  observer  /  expenmenter  selectively  provide  instructions,  ask 
questions  over  net  to  players  or  observers  or  controllers?  If  not 
can  this  be  added? 
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3.1 .  Ability  to  capture  physical  actions: 


3.1.1.  Movement  in  OR 


3.1.2.  Workstation 


3.1. 1.1.  Keystrokes 


3.1. 1.2.  Screen  navigation 


3. 1. 1.3.  Comm  buttons/pedals 


3.1.3.  Other  devices 


3.2.  Ability  to  capture  information  exchange 


3.2.1.  Written 


3.2.2.  Verbal 


3.2.3.  Visual 


3.3.  Capability  of  recording  contents  of  CCS  Display 


3.4.  Ability  to  capture  timing  of  actions  and  other  events  to 
required  accuracy 


3.5.  Availability  of  observers  as  assessors. 


3.6.  Ability  to  capture  data  in  post  analysis  (by  re-running 
scenario) 


~  ~  >  l _ I _ i _ *./  _ itii; _  4.  - 


3.7.  Existence  of  training  MOPs 


e.g  video,  movement  sensors  in  Ops  Room  or  on  person. 


Quad  video,  software  based? 


Quad  video,  software  based? 


On  net,  off  net? 


Lap  tops,  PDAs,  key  pads,  etc. 


Non  CCS:  formal  (msg  pad,  format),  informal  (post  it  note,  etc). 
Timing,  content,  source,  addressee(s) 


Network  vs  non  network 

Timing,  content,  source,  addressee(s) 


Face  to  face,  stateboards,  non  CCS  info  sources  (e.g.  GCCS,  plot, 
manuals,  charts,  etc) 

Timing,  content,  addressee(s)  i.e.  what  is  source  of  information. 


Time  line,  screen  dump  vs  data  content. 


Timeline  resolution,  registration  to  other  workstations,  etc. 


Instructor  SMEs  to  act  as  observers  and  interpreters  for  real  time  or 
post  hoc  review. 


Can  responses  to  scenano  be  re-run?  How  sophisticated, 
comprehensive  is  this  capability?  Can  tempo  and/or  content  be 
modified  for  each  re-run.  Can  data  sets  be  aggregated? 


Which  Ops  Room  positions?  Doc  available. 


4.1 .  Ability  to  store  data  in  a  format  that  is  readily  exported 
for  analysis 

Can  data  be  used  in  other  analysis  display  software  applications? 

4.2.  Ability  to  analyse  data 

Existing  or  on  -site  data  analysis  capabilities? 

4.3.  Ability  to  display  data 

Existing  or  on  -site  data  display  capabilities? 

4.4.  Reliable  data  storage 

Types  of  reliability  problems 

Crash  risk 

Data  formats 

Compatibility  with 

4.5.  Access  to  data  after  collection 

Can  data  be  taken  off  -site? 

Secunty  issues? 

5.1.  Objective 

Time  and  accuracy  based  metrics  excluding  rating  scales 

5  2.  Subjective 

Subjective  instructor  based  (criteria,  rating  scales,  etc). 

5.3.  Machine-based 

Automatic  implementation  of  objective  measures  above? 

6.1.  Interruption  free 

Unbroken  sequences?  Intrusive  observers? 

6.2.  Appropnate  illumination 

Colour,  level 

6.3.  Appropriate  sound  levels 

Weapons,  engine  noise,  talk 

6  4.  Appropriate  Ship  motion 

Changes  in  direction,  interference  with  movement,  sickness? 

6.5.  Appropnate  clothing 

Is  clothing  representative  (hoods,  gloves,  heat,) 

6.6.  Layout  appropnate  to  Ops  Room(where  required) 

Lines  of  sight  to  stateboards,  space  for  notes,  etc. 

6.7.  Range  of  tasks 

Can  all  tasks  be  simulated?  What  tasks  cannot? 

6.8.  Tempo  of  operations 

Same  pace  as  operations? 
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e.g.  volume  and  type  of  text  messages,  responsiveness,  sound, 
level  of  radio  comms,  etc. 
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C.l  ORTT 


;*}>*  -  ^  \  N  / ;  '  r&  v  '  ***'  ,\'*>,  . \  w. 

1.1.  Availability  of  facility 

Booking  process  (advance  notice  etc) 

Program  (annual?) 

Sharing  vs  exclusive 

Duration  of  booking  24/7,  etc. 

ORTT:  (16  hours  day)  for  20001/2002;  8  hours  day  training/  8  hours  L-M  -5  days 
week. 

Must  be  booked  by  March-Apnl  for  next  year  schedules  Starts  scheduling  in  March 
-done  by  September 

Training  given  first  priority:  blocks  out,  March,  Apnl,  Sept,  Oct  for  CFNOS.  Also 
vacations. 

Pnonty  with  LM  determined  in  advance.  In  future  LM  needs  will  reduce. 

Immediate  use  is  extensive  for  shakedown  and  building  scenarios  ORTT  priority  is 
Megaphase,  Mariants,  OTTs  (4x5days),  sub-team  training, 

Equipment  is  available  dunng  spare  time  in  megaphase,  but  not  staff  for  running  the 
trainers;  TTC  staff  are  needed  for  scenarios  logistics.  Needs  would  be  reduced  if 
only  looking  at  one  warfare  area. 

Facility  is  now  30%  overbooked. 

Could  fit  in  a  3  day  pilot  -  some  time  this  year;  air  warfare  side  is  most  congested 
right  now. 

Longer  blocks  available  next  year. 

Notes: 

Data  would  be  available  from  existing  training  courses  to  do  baseline. 

NCOT  will  reduce  some  of  this  load  when  it  fully  comes  on  line  because  it  does 
some  of  the  basic  equipment  familiarisation. 

NCOT  -  lots  of  free  time  now  -  but  wll  take  over  ORTT/CSTC  capability  in  future 
and  should  free  up  time.  However,  this  excess  capability  is  likely  to  be  absorbed  by 
existing  Navy  needs. 

1.2.  Set  up  time  for  facility 

How  much  work  /  time  to  prepare  from  current  status. 

Half  day,  one  hour  for  existing  scenarios 

1.3.  Number  of  CCS  workstations 

All  linked? 

Full  OR  less  CANEWS,  CANTASS  GMCS. 

10  workstations  in  each  of  two  rooms  tbndge  (only  used  as  a  simulation  input) 

1.4.  OR  Team  simulated 

0P/SUP/WD/0R0/C0 

Specialties  (EW/SONAR/ETC) 

Outside  sources  (bridge,  helo,  etc) 

CO,  ORO,  SWIC,  ASWC,  ASAC.RT1  ,RT2,  TrackSup,  EWS,  SCS,  Bridge 

1.5.  Suitable  system  docs  &  manuals  available? 

Availability:  what,  where,  removable? 

Suitability:  Contents,  Set  up  vs  maintenance. 

Yes  on  CD. 
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1 .6  Availability  of  Technical/Controller  support  for : 

1.6.1.  Software/hardware  during  preparation 

Help  with  set  up  of  equipment 

Scenano  development  and  set  up 

Set  data  capture  equipment,  software,  analysis 

Yes  -depends  on  schedule  -  and  length. 

One  game  takes  one  month  to  develop  from  scratch.  But  existing  scenarios  can  be 
readily  implemented.  Most  of  the  scenanos  will  be  available  within  a  year.  Many  of 
the  gaming  scenarios  are  available  now  (e.g.  ‘clockwork  mouse”) 

1 .6.2.  Software/hardware  during  testing 

Maintenance  /  repair  of  crashes 

Ad  hoc  changes  to  scenano  (Unplanned  insertion  of  data) 

Yes 

Yes  (but  event  must  be  in  database) 

1.6.3.  Control  scenario  play 

Availability  of  controller  staff  as  scenario  players,  war  game 
directors? 

Yes  -depends  upon  schedule  5  days/0800-1600 

1.6.4.  Software/hardware  dunng  analysis 

Data  capture  and  analysis,  downloading,  configuring  for  other 
analysis  software,  etc. 

Analysis  after  hours  -  would  need  to  be  done  by  LM 

1.65.  Trg  of  HSI®  test  personnel  in  technical 

Issues 

How  are  HSI®™  staff  to  be  trained 

Trained  for  audio-visual  -  on  the  job  for  data  insertion 

Yes. 

1 .7.  Availability  of  Personnel  as  test  participants 

TBD 

171.  Number 

How  many  can  we  get? 

1.7  2.  Experience  /  qualifications 

What  range  of  expenence  and  qualification  (e.g.  students  only?) 

1.7.3.  Individuals  or  teams 

Will  they  have  worked  together  before? 

1.7.4.  Notice  required 

How  far  ahead  is  booking  required’ 

1  7  5.  Range  of  OR  team  represented 

Different  availability  for  different  levels?  Are  “filler"  personnel 
available  for  part  task  simulation? 

1 .8.  Video  recording  available 

Yes -from  2  cameras 

18.1.  Type  of  video 

#  workstations  covered, 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources. 

Macro  coverage  /  resolution  of  CCS  screen  data 

Tracking  (pan,  zoom,  etc) 

Low  light  /  red  light 

Global  view  of  all  workstations  with  wide  area  coverage 

Length  of  scenano 

Yes 

No  (this  is  captured  by  software) 

No 

Yes 

1 .8.2.  Eye  tracking  capability 

Video, 

other 

Yes- but  independent 

1 .9.  Audio  recording  available 

1.91.  Type  of  audio 

#  workstations  covered  (internal  vs  external  to  OR), 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources, 
identification  of  position(s),  incoming  vs  outgoing 
audio  quality  (noise,  etc) 
off  net  speech  in  OR 

2  mikes  for  general  audio  (part  of  audio)  -  some  individual  voices  could  be 
captured-  if  the  area  is  quiet. 

Captures  all  internal  net  comms  both  ops  and  sim  co-ordinated  on  time  base. 

Comms  data  are  associated  with  each  console  position 
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1.10.  Information  exchange  (is  it  represented? 


1.10.1.  Internal  net  in  OR 


1 .1 0.2  External  nets  to  OR  on  ship 


1.10.3.  External  nets  to  ship 


1.10.4.  Text 


1.10.5.  Face  to  face  in  OR 


1.10.6.  Other  information  exchanges 


1.11.  Customise  environment  for  data  collection 


1.11.1.  Extra  audio  and  video 


1 .1 1 .2.  Add  on  software  modules 


1.12.  Standalone  keypads,  notebooks  or  PDA's  for  data 
entry  by  participants  or  observers? 


1.13  Reliability  of  system 


1.14  Space  for  observers 


Who  can  be  on  it  in  trainer? 


Who  can  be  on  it  trainer? 

Who  can  be  on  it  trainer? 

What  types  of  text  messages? 
Level  of  text  message  activity? 


Direct  voice  contact  in  OR? 


State  boards 

Manuals  /  Charts  /  T acpacs  /  Post  its  /  etc. 


Ability  to  add  COMDAT  prototype  equipment  /  software 
Technical  support  for  this? 


Add  extra  audio  /  video  data  capture  devices 


Ability  to  add  on  software  modules  to  inteiject  data  into  canned 
scenarios/simulations 


Eg  for  workload  ratings  at  intervals 
Or  for  response  to  freeze  probes 
or  for  diary  entry 


Types  of  reliability  problems 
Crash  risk 

Repair  time  /  typical  down  time 
Repeatability  of  scenanos 


Is  there  space  for  more  people  to  observe  /  capture  data. 
Worst  case  =  one  observer  at  each  woik  station. 


Yes:  would  need  either  physical  handoff  or  sent  through  CCS 


Only  if  ops  room  quiet  -difficult  to  implement  for  ORS  but  might  be  acceptable  for 
ORO,  either  on  system  mike  or  room  mikes 


Yes 

Manuals-Y,  Charts-Y,  Tacpacs-Y,  Post  its-Y 


No,  only  as  simulation  and  this  would  need  to  be  contracted  with  LM 


Would  need  to  be  contracted  with  LM 


Yes  -  for  dedicated  tnals  -  not  dunng  training 
Yes 

Unpredictable,  since  system  is  still  in  eariy  days 

Can  do  workarounds  -  may  need  to  add  25%  extra  capacity  to  allow  for  this. 

Yes 


2. 1 .  Environmental  control 

(lighting,  acoustics,  thermal,  motion) 


2.2.  Pre-packaged  scenario 

2.3.  Complexity  of  scenanos 


2.3.1.  Mission  types 


2.3.2.  Mission  segments 

2.3.3.  Number  of  players 


Adjustability  and  control  over  lighting 

Adjustability  and  control  over  acoustics 

Yes 

N  -  but  representative;  warmer-no  external  noise  eg  weapons 

Availability? 

Modifiabilty?  (e.g.  freeze  and  tracking  probes) 

Yes 

Tracking  yes-Freeze-tracking  probes  OK  by  voice  or  CCS 

Location 
Threat  type 
TG  configuration 
OOTW,  Aid  civil  power,  etc 


Part  task  breakdown  capability 


Multiple  players  on  single  task,  or  single  player  multiple  roles. 


Not  now -could  be  done 
Anything 

Anything  including  other  nations 
Yes 


Max  is  2. 3  in  the  case  of  Air.  Front  4  are  interchangeable. 
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2  3.4.  Number  and  types  of  assets 

How  many  and  what  types  en  /  f  r  /  neutral  ships,  a/c,  wpns, 
sensors,  etc. 

Can  these  be  varied? 

Yes  all  CPF 

2  3.5.  Number  and  levels  of  team  roles 

Can  these  be  vaned? 

Yes 

2.4.  Ability  to  run  pre-packaged  scenario  to  a 

defined  timeline 

Certain  events  at  certain  points  on  time  line,  regardless  of  player 
response. 

Data  to  be  collected  on  that  time  line. 

Yes 

2.5.  Ability  to  control  select  variables  in  real  time 

Ad  hoc  failures  of  Circuit  or  Net,  workstation  to  fail,  wpn  system  to 
fail. 

Yes 

2.6.  Ability  to  interject  probes  In  real  time 

Freeze  probes 

Tracking  probes 

Yes 

2.7.  Ability  to  stop/re-start  scenanos 

Yes 

2  8.  Ability  to  backup/re-run  scenarios 

Yes 

2  9.  Ability  to  dynamically  modify  scenano 

depending  upon  responses 

If  x  then  y  e.g.  if  fails  to  respond  to  threat,  re-insert  same  threat 
later 

If  a  then  b 

Yes,  but  would  need  to  be  continuous  manual  control 

2 10.  Comms  among  expenmenters  /  controllers 

Can  observer  /  experimenter  selectively  provide  instructions,  ask 
questions  over  net  to  players  or  observers  or  controllers?  If  not 
can  this  be  added? 

Yes 

4B§  -  W  v 
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3. 1 .  Ability  to  capture  physical  actions: 

3.1.1.  Movement  in  OR 

e.g  video,  movement  sensors  in  OR  or  on  person. 

Yes 

3.1,2.  Workstation 

3.12.1.  Keystrokes 

Quad  video,  software  based? 

Yes 

3.1. 2.2.  Screen  navigation 

Quad  video,  software  bsed? 

Yes 

3. 1.2.3.  Comm  buttons/pedals 

On  net,  off  net? 

Yes 

3.1.3  Other  devices 

Lap  tops,  PDAs,  key  pads,  etc. 

Yes 

3  2  Ability  to  capture  information  exchange 

3  2.1.  Written 

Non  CCS:  formal  (msg  pad,  format),  informal  (post  it  note,  etc). 
Timing,  content,  source,  addressee(s) 

Yes  -tied  to  video.  If  message,  then  would  go  into  file. 

Informal  exchanges  -  no,  author  would  need  to  indicate  position  and  time  of 
message  manually. 

3  2.2.  Verbal 

Network  vs  non  network 

Timing,  content,  source,  addressee(s) 

Yes 
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3.2.3.  Visual 

Face  to  face,  stateboards,  non  CCS  info  sources  (e.g.  GCCS,  plot, 
manuals,  charts,  etc) 

Timing,  content,  addressee(s)  i.e.  what  is  source  of  information. 

Yes 

3.3.  Capability  of  recording  contents  of  CCS  Display 

Time  line,  screen  dump  vs  data  content. 

Screen  saved  in  software 

3.4  Ability  to  capture  timing  of  actions  and  other  events 
to  required  accuracy 

Timeline  resolution,  registration  to  other  workstations,  etc. 

Timeline  -  yes:  -possibly  1  second-  screen  shows  to  1/100. 

3.5.  Availability  of  observers  as  assessors. 

Instructor  SMEs  to  act  as  observers  and  interpreters  for  real  time 
or  post  hoc  review. 

Yes-  except  during  megaphase-  depends  on  availability  of  local  talent. 

3.6.  Ability  to  capture  data  in  post  analysis  (by  re-running 
scenario) 

Can  responses  to  scenario  be  re-run? 

How  sophisticated,  comprehensive  is  this  capability?  Can  tempo 
and/or  content  be  modified  for  each  re-run.  Can  data  sets  be 
aggregated? 

Yes 

High 

Yes:  fast  to  1 0x  or  slow  motion  to  1/4 

No 

3.7.  Existence  of  training  MOPs 

Which  OR  positions?  Doc  available. 

Yes 

4.1 .  Ability  to  store  data  in  a  format  that  is  readily 
exported  for  analysis 

Can  data  be  used  in  other  analysis  display  software  applications? 

No 

4.2.  Ability  to  analyse  data 

Existing  or  on  -site  data  analysis  capabilities? 

Yes,  playback  on  site 

4.3.  Ability  to  display  data 

Existing  or  on  -site  data  display  capabilities? 

Yes,  playback  on  site 

4.4.  Reliable  data  storage 

Types  of  reliability  problems 

Crash  nsk 

Data  formats 

Compatibility  with 

No  standard  data  formats 

Little  crash  risk 

4.5  Access  to  data  after  collection 

Can  data  be  taken  off  -site? 

Secunty  issues? 

No 

No 

SjOSiaErl 


5.1.  Objective 


5.2.  Subjective 


5.3.  Machine-based 


6.1.  Interruption  free 

6.2.  Appropriate  illumination 


Time  and  accuracy  based  metrics  excluding  rating  scales 


Subjective  instructor  based  (criteria,  rating  scales,  etc). 


Automatic  implementation  of  objective  measures  above? 


Unbroken  sequences?  Intrusive  observers? 

Yes 

Colour,  level 

Y  -low  white 
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6  3  Appropriate  sound  levels 

Weapons,  engine  noise,  talk 

No 

6.4.  Appropriate  Ship  motion 

Changes  in  direction,  interference  with  movement,  sickness? 

No 

6.5.  Appropriate  clothing 

Is  clothing  representative  (hoods,  gloves,  heat,) 

Yes 

6  6.  Layout  appropriate  to  OR  (where  required) 

Lines  of  sight  to  stateboards,  space  for  notes,  etc. 

Yes 

6.7.  Range  of  tasks 

Can  all  tasks  be  simulated?  What  tasks  cannot? 

Yes 

6.8.  Tempo  of  operations 

Same  pace  as  operations? 

Yes 

6.9.  Communication 

e  g.  volume  and  type  of  text  messages,  responsiveness,  sound, 
level  of  radio  comms,  etc. 

Lower  volume  of  text  messages  -  lower  level  of  voice  comm  than  in  real  TG 

6.10  Complexity 

e  g.  multi-tasking 

Yes.  At  ORO  level  no  Damcat,  logistics  for  injured  personnel 

6.11.  Workload 

Is  workload  representative 

Yes- see  above 

612  Duration 

Sustained  operations,  time  of  day,  fatigue 

Yes  -  more  easily  done  on  ship,  but  typical  scenanos  are  less  than  4hrs 

613  Manning  levels 

Realistic  for  operation 

No  CSE  -  no .  no  extra  personnel  for  action  stations 

6.14.  Threat  levels 

Are  they  realistic? 

Yes 

6.15.  Force  configuration  and  role. 

Enemy  and  friendly. 

Yes 

616  Stress 

Type,  intensity 

Yes 

6.17.  M 

ission  (s) 

e.g  OOTW  players.  Multi-national,  Aid  to  civil  power. 

Can  be  done 

6.18.  S 

ocial  dimensions 

Use  of  intact  teams? 

Yes 

6.19.  0 

ther 

Quals  /  experience  of  participants? 
knowledge  of  mission  pnor  to  start  scenario. 

None  experienced 

Y 
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1.1.  Availability  of  facility 

Booking  process  (advance  notice  etc) 

Program  (annual?) 

Sharing  vs  exclusive 

Duration  of  booking  24/7,  etc. 

SameasORTT 

Currently  more  available  than  ORTT 

Must  be  shared  with  MD  in  the  short  term 

1.2.  Set  up  time  for  facility 

How  much  woik  /  time  to  prepare  from  current  status. 

Less  than  1  day 

1 .3.  Number  of  CCS  workstations 

All  linked? 

38  WS  in  two  classrooms/all  linked. 

1.4.  OR  Team  simulated 

OP/SUP/WD/ORO/CO 

Specialities  (EW/SONAR/ETC) 

Outside  sources  (bride,  helo,  etc) 

All+CANEWS;  no  bridge  WS  but  can  provide  info  from  bndge  and  helo 

1 .5.  Suitable  system  docs  &  manuals  available? 

Availability:  what,  where,  removable? 

Suitability:  Contents,  Set  up  vs  maintenance. 

CD  -  both  operating  and  maintenance 

1 .6.  Availability  of  Technical/Controller  support  for 

1.6.1.  Software/hardware  during  preparation 

Help  with  set  up  of  equipment 

Scenano  development  and  set  up 

Set  data  capture  equipment,  software,  analysis 

Yes  through  arrangements  with  MD.  Possibility  of  training  T&E  staff  in  procedures. 

1.6.2.  Software/hardware  during  testing 

Maintenance  /  repair  of  crashes 

Ad  hoc  changes  to  scenano  (Unplanned  insertion  of  data) 

Yes  -  but  limited.  No  available  Navy  infrastructure  at  present  for  game  players. 

1 .6.3.  Control  scenano  play 

Availability  of  controller  staff  as  scenario  players,  war  game 
directors? 

No  specifically  assigned  Navy  staff  for  this  responsibility. 

1 .6.4.  Software/hardware  during  analysis 

Data  capture  and  analysis,  downloading,  configuring  for  other 
analysis  software,  etc. 

No,  but  could  be  developed  by  MD 

1 .6.5.  Trg  of  HS1®  test  personnel  in  technical  issues 

How  are  HS®™  staff  to  be  trained 

Yes,  on  site 

1 .7.  Availability  of  Personnel  as  test  participants 

TBD 

1.7.1.  Number 

How  many  can  we  get? 

1.72.  Expenence/ qualifications 

What  range  of  expenence  and  qualification  (e.g.  students 
only?) 

1.7.3.  Individuals  or  teams 

Will  they  have  worked  together  before? 

1.7.4.  Notice  required 

How  far  ahead  is  booking  required? 

1.7.5.  Range  of  OR  team  represented 

Different  availability  for  different  levels?  Are  “filler”  personnel 
available  for  part  task  simulation? 

1 .8.  Video  recording  available 
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1  8.2.  Eye  tracking  capability 
1 .9.  Audio  recording  available 


1.9.1.  Type  of  audio 


1.10.  Information  exchange  (is  it  represented? 


1.101  Internal  net  in  OR 


1.10.2.  External  nets  to  OR  on  ship 


1.103.  External  nets  to  ship 


1.104.  Text 


1.10.5.  Face  to  face  in  OR 


1106.  Other  information  exchanges 


1.11  Customise  environment  for  data  collection 


1.11.1.  Extra  audio  and  video 


1 .1 1 .2.  Add  on  software  modules 


1.12.  Standalone  keypads,  notebooks  or  PDA's  for  data 
entry  by  participants  or  observers? 


1.13.  Reliability  of  system 


1,14.  Space  for  observers 


#  workstations  covered, 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources. 
Wide  area  coverage 

Macro  coverage  /  resolution  of  CCS  screen  data 
Quad  coverage 
Tracking  (pan,  zoom,  etc) 

Low  light  /  red  light 


Video,  other 


#  workstations  covered  (internal  vs  external  to  OR), 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources, 
identification  of  position(s),  incoming  vs  outgoing 
audio  quality  (noise,  etc) 
off  net  speech  in  OR 


Same  as  ORTT 


Yes,  linked  to  scenano 
Good  (on  the  word  of  the  MD  rep) 

Need  extra  audio  recording  -  not  co-ordinated 


No  external  sim  room  for  providing  this  capability 


Who  can  be  on  it  in  trainer? 


Who  can  be  on  it  trainer? 


Who  can  be  on  it  trainer? 


What  types  of  text  messages? 
Level  of  text  message  activity? 


Direct  voice  contact  in  OR? 


State  boards 

Manuals  /  Charts  /  Tacpacs  /  Post  its  /  etc. 


Ability  to  add  COMDAT  prototype  equipment  /  software 
Technical  support  for  this? 


Add  extra  audio  /  video  data  capture  devices 


Ability  to  add  on  software  modules  to  interject  data  into  canned 
scenanos/simulations 


Eg  for  workload  ratings  at  intervals 
Or  for  response  to  freeze  probes 
or  for  diary  entry 


Types  of  reliability  problems 
Crash  nsk 

Repair  time  /  typical  down  time 
Repeatability  of  scenarios 


Is  there  space  for  more  people  to  observe  /  capture  data. 
Worst  case  =  one  observer  at  each  work  station. 


Yes 


Yes 

Satisfactory 


Yes,  would  need  separate  audio 


No,  could  be  added 
No,  could  be  added 


No,  except  as  simulated  and  programmed  by  MDA 


Unknown 
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2.1.  Environmental  control 

(lighting,  acoustics,  thermal,  motion) 

Adjustability  and  control  over  lighting 

Adjustability  and  control  over  acoustics 

Lighting  vanable 

No 

2.2.  Pre-packaged  scenario 

Availability? 

Modifiabilty?  (e.g.  freeze  and  tracking  probes) 

Yes 

Yes 

2.3.  Complexity  of  scenanos 

2.3.1.  Mission  types 

Location,  threat  type,  TG  configuration,  OOTW,  Aid  civil  power, 
etc 

Any 

23.2.  Mission  segments 

Part  task  breakdown  capability 

Yes 

2.3.3.  Number  of  players 

Multiple  players  on  single  task,  or  single  player  multiple  roles. 

Both  -  very  flexible  groupings  of  controllers  and  players 

2.3.4.  Number  and  types  of  assets 

How  many  and  what  types  en  /  f  r  /  neutral  ships,  a/c,  wpns, 
sensors,  etc. 

Can  these  be  vaned? 

No  limit 

Yes 

2.3.5.  Number  and  levels  of  team  roles 

Can  these  be  varied? 

Yes 

2.4.  Ability  to  run  pre-packaged  scenano  to  a 

defined  timeline 

Certain  events  at  certain  points  on  time  line,  regardless  of 
player  response. 

Data  to  be  collected  on  that  time  line. 

Yes 

Yes,  more  could  be  programmed 

2.5.  Ability  to  control  select  variables  in  real  time 

Ad  hoc  failures  of  Circuit  or  Net,  workstation  to  fail,  wpn  system 
to  fail. 

Yes 

2.6.  Ability  to  interject  probes  in  real  time 

Freeze  probes 

Tracking  probes 

Yes 

Yes 

2.7.  Ability  to  stop/re-start  scenarios 

Yes 

2.8.  Ability  to  backup/re-run  scenarios 

No 

2.9.  Ability  to  dynamically  modify  scenario 

depending  upon  responses 

If  x  then  y  e.g.  if  fails  to  respond  to  threat,  re-insert  same  threat 
later 

If  a  then  b 

Yes,  through  controller 

2.10.  Comms  among  expenmenters  /  controllers 

Can  observer  /  expr  provide  instructions,  ask  questions  over 
net  to  players  or  observers  or  controllers?  If  not  can  this  be 
added? 

Yes 

itH f {^r1  - |§p,  H  W  . : ** ^j|l 
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3.1 .  Ability  to  capture  physical  actions: 

3.1.1.  Movement  in  OR 

e.g  video,  movement  sensors  in  OR  or  on  person. 

Yes,  but  extra  video  required 

3.1.2.  Workstation 
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3.1. 2.1.  Keystrokes 

Quad  video,  software  based? 

Yes  -  software 

3. 1.2.2.  Screen  navigation 

Quad  video,  software  bsed? 

Yes-  software 

3. 1.2.3.  Comm  buttons/pedals 

On  net,  off  net? 

Yes-  software 

3.1  3.  Other  devices 

Lap  tops,  PDAs,  key  pads,  etc. 

Yes,  not  co-ordinated 

3.2.  Ability  to  capture  information  exchange 

3.2.1.  Wntten 

Non  CCS:  formal  (msg  pad,  format),  informal  (post  it  note,  etc). 
Timing,  content,  source,  addressee(s) 

Yes 

3.2.2.  Verbal 

Network  vs  non  network 

Timing,  content,  source,  addressee(s) 

Yes,  need  extra  recorder  for  non-network 

3.2.3.  Visual 

Face  to  face,  stateboards,  non  CCS  info  sources  (e.g.  GCCS, 
plot,  manuals,  charts,  etc) 

Timing,  content,  addressee(s)  i.e.  what  is  source  of  information. 

Yes,  need  extra  recorder  for  non-network 

3  3.  Capability  of  recording  contents  of  CCS  Display 

Time  line,  screen  dump  vs  data  content. 

Yes  to  all 

3.4.  Ability  to  capture  timing  of  actions  and  other 
events  to  required  accuracy 

Timeline  resolution,  registration  to  other  workstations,  etc. 

Less  <  sec 

3.5.  Availability  of  observers  as  assessors. 

Instructor  SMEs  to  act  as  observers  and  interpreters  for  real 
time  or  post  hoc  review. 

Same  as  ORTT 

3.6.  Ability  to  capture  data  in  post  analysis  (by  re¬ 
running  scenano) 

Can  responses  to  scenano  be  re-mn? 

How  sophisticated,  comprehensive  is  this  capability?  Can 
tempo  and/or  content  be  modified  for  each  re-run.  Can  data 
sets  be  aggregated? 

Yes 

Very 

Content  yes,  tempo  no. 

No,  would  require  special  software 

3.7.  Existence  of  training  MOPs 

Which  OR  positions?  Doc  available. 

See  ORTT 

awiggg*  *#.  *s§  ->9'  i 
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4.1 .  Ability  to  store  data  in  a  format  that  is  readily 
exported  for  analysis 

Can  data  be  used  in  other  analysis  display  software 
applications? 

No,  would  require  extra  software 

4.2.  Ability  to  analyse  data 

Existing  or  on  -site  data  analysis  capabilities? 

Yes,  manual  analysis,  MD  could  develop  required  software 

4.3.  Ability  to  display  data 

Existing  or  on  -site  data  display  capabilities? 

Yes 

4.4.  Reliable  data  storage 

Types  of  reliability  problems 

Crash  risk 

Data  formats 

Compatibility  with 

Yes 

No 

4.5  Access  to  data  after  collection 

Can  data  be  taken  off  -site? 

Secunty  issues? 

No,-  would  need  special  arrangements  and  local  playback  capability. 

No  different  from  ORTT,  CSTC  -  commercial  propnetary  issues 
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5.1.  Objective 

Time  and  accuracy  based  metrics  excluding  rating  scales 

5.2.  Subjective 

Subjective  instructor  based  (cntena,  rating  scales,  etc). 

5.3.  Machine-based 

Automatic  implementation  of  objective  measures  above? 

iMErtSr. . .  ■ ,  -  ,w-  s 

1 

6.1.  Interruption  free 

Unbroken  sequences? 

Intrusive  observers? 

Yes. 

Yes;  no  remote  observation  right  now  but  could  be  added 

6.2.  Appropnate  illumination 

Colour,  level 

Room  lighting  -  not  variable 

6.3.  Appropnate  sound  levels 

Weapons,  engine  noise,  talk 

No -except  for  talk 

6.4.  Appropnate  Ship  motion 

Changes  in  direction,  interference  with  movement,  sickness? 

No 

6.5  Appropnate  clothing 

Is  clothing  representative  (hoods,  gloves,  heat,) 

Could  be 

6  6.  Layout  appropnate  to  OR  (where  required) 

Lines  of  sight  to  stateboards,  space  for  notes,  etc. 

No,  but  could  be  adapted 

6  7.  Range  of  tasks 

Can  all  tasks  be  simulated?  What  tasks  cannot? 

Yes,  except  GMCS 

6.8.  Tempo  of  operations 

Same  pace  as  operations? 

Could  be 

6.9.  Communication 

e.g.  volume  and  type  of  text  messages,  responsiveness,  sound, 
level  of  radio  comms,  etc. 

Yes,  depends  on  configuration,  but  generally  less 

6.10.  Complexity 

e.g.  multi-tasking 

Less  than  OR 

6.11.  Workload 

Is  workload  representative 

Depends  on  position  -  probably  acceptable  at  the  Operator  level,  unknown  and 
possibly  doubtful  at  the  command  level 

6.12.  Duration 

Sustained  operations,  time  of  day,  fatigue. 

Possible  -  limited  by  2GB  of  data  storage 

6.13.  Manning  levels 

Realistic  for  operation 

No 

6.14.  Threat  levels 

Are  they  realistic? 

Yes 

6. 1 5  Force  configuration  and  role. 

Enemy  and  fnendly. 

Yes 

6.16.  Stress 

Type,  intensity. 

No 

6.17,  Mission  (s) 

e.g.  OOTW  players.  Multi-national,  Aid  to  civil  power. 

Yes,  any  possible 

6.18.  Social  dimensions 

Use  of  intact  teams? 

Yes 

6.19.  Other 

Quals  /  expenence  of  participants?  Background  knowledge  of 
mission  prior  to  start  scenario. 

Same  as  ORTT  -depends  on  choice  and  availability. 
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C3.  CSTC 


1,1.  Availability  of  facility 

Booking  process  (advance  notice  etc) 

Program  (annual?) 

Sharing  vs  exclusive 

Duration  of  booking  24/7,  etc. 

Same  as  ORTT 

More  available:  shifts  of  8  hours, 24/7  operation. 

LM  8hrs,  CFNOS8hrs 

Very  little  available  capacity. 

Booking  procedure  same  as  ORTT. 

1.2.  Set  up  time  for  facility 

How  much  work  /  time  to  prepare  from  current  status. 

One  hour 

1 .3.  Number  of  CCS  workstations 

All  linked? 

1 1  linked  plus  out-stations  in  OR 

1  4.  OR  Team  simulated 

OP/SUP/WD/ORO/CO 

Specialities  (EW/SONAR/ETC) 

Outside  sources  (bridge,  helo,  etc) 

All  plus  CANEWS,  HMS,  SPS;  no  Tail,  no  GMCS. 

1 .5.  Suitable  system  docs  &  manuals  available? 

Availability  what,  where,  removable? 

Suitability:  Contents,  Set  up  vs  maintenance. 

Y:  no  user  manual-could  be  on  disk. 

1.6.  Availability  of  Technical/Controller  support  for 

1.6.1.  Software/hardware  during  preparation 

Help  with  set  up  of  equipment 

Scenario  development  and  set  up 

Set  data  capture  equipment,  software,  analysis 

Yes 

1  6  2.  Software/hardware  dunng  testing 

Maintenance  /  repair  of  crashes 

Ad  hoc  changes  to  scenario  (Unplanned  insertion  of  data) 

Yes 

1  6.3.  Control  scenano  play 

Availability  of  controller  staff  as  scenano  players,  war  game 
directors? 

Yes 

1  6.4  Software/hardware  dunng  analysis 

Data  capture  and  analysis,  downloading,  configuring  for  other 
analysis  software,  etc. 

Limited:  recording.  Can  only  playback  in  CSTC; 

No  time  tags,  have  to  remember  time  of  events. 

There  is  a  trade-off  between  amount  of  information  recorded  and  playback  time,  could 
be  as  little  as  10  mins  for  everything. 

1 .6.5.  Trg  of  HSKD  test  personnel  in  technical  issues 

How  are  HSKD™  staff  to  be  trained 

Yes,  easier  than  ORTT. 

1 .7.  Availability  of  Personnel  as  test  participants 

1.71.  Number 

How  many  can  we  get? 

1 .7.2.  Expenence  /  qualifications 

What  range  of  experience  and  qualification  (e.g.  students 
only?) 

1.7.3  Individuals  or  teams 

Will  they  have  worked  together  before? 

1.7.4.  Notice  required 

How  far  ahead  is  booking  required? 

1 .7.5.  Range  of  OR  team  represented 

Different  availability  for  different  levels?  Are  “filler”  personnel 
available  for  part  task  simulation? 

1 .8.  Video  recording  available 
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1.8.1.  Type  of  video 

♦  workstations  covered, 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources. 

Wide  area  coverage 

Macro  coverage  /  resolution  of  CCS  screen  data 

Quad  coverage 

Tracking  (pan,  zoom,  etc) 

Low  light  /red  light 

None  -  but  could  be  added  eg  C-SAVRAS 

1.8.2.  Eye  tracking  capability 

Video, 

other 

No 

1 .9.  Audio  recording  available 

1.9.1.  Type  of  audio 

#  workstations  covered  (internal  vs  external  to  OR), 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources, 
identification  of  position(s),  incoming  vs  outgoing 
audio  quality  (noise,  etc) 
off  net  speech  in  OR 

No,  but  the  capability  is  there.  Passive  audio  tape  could  be  implemented  but  would 
not  be  co-ordinated  with  software  data  capture.  C-SAVRAS  may  also  be  plugged  into 
any  6  ws. 

Off-network  audio  would  require  manual  recording. 

Quality  good 

1.10.  Information  exchange  (is  it  represented? 

1.10.1.  Internal  net  in  OR 

Who  can  be  on  it  in  trainer? 

Yes  plus  PRC  walkie  talkie.  All  inputs  are  put  in  real  time,  cannot  be  pre-programmed 

1.10.2.  External  nets  to  OR  on  ship 

Who  can  be  on  it  trainer? 

Yes 

110.3.  External  nets  to  ship 

Who  can  be  on  it  trainer? 

Yes 

1.10.4.  Text 

What  types  of  text  messages? 

Level  of  text  message  activity? 

Yes,  but  manually  handled 

1.10.5.  Face  to  face  in  OR 

Direct  voice  contact  in  OR? 

Yes 

1 .10.6.  Other  information  exchanges 

State  boards 

Manuals  /  Charts  /  Tacpacs  /  Post  its  /  etc. 

Yes 

Yes 

1.11.  Customise  environment  for  data  collection 

Ability  to  add  COMDAT  prototype  equipment  /  software 

Technical  support  for  this? 

Yes-if  configured  for  ship,  then  would  fit  CSTC. 

Yes 

1.11.1.  Extra  audio  and  video 

Add  extra  audio  /  video  data  capture  devices 

Yes 

1 .1 1 .2.  Add  on  software  modules 

Ability  to  add  on  software  modules  to  interject  data  into  canned 
scenanos/simulations 

No-  this  would  need  to  be  done  manually 

1.12.  Standalone  keypads,  notebooks  or  PDA's  for  data 
entry  by  participants  or  observers? 

Eg  for  workload  ratings  at  intervals 

Or  for  response  to  freeze  probes 
or  for  diary  entry 

Yes 

1.13.  Reliability  of  system 

Types  of  reliability  problems 

Repair  time  /  typical  down  time 

Repeatability  of  scenanos 

90%+ 

Hour 

High 

1.14.  Space  for  observers 

Is  there  space  for  more  people  to  observe  /  capture  data. 

Worst  case  =  one  observer  at  each  workstation. 

Yes 
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2.1.  Environmental  control 

(lighting,  acoustics,  thermal,  motion) 

2.2.  Pre-packaged  scenario 


2.3.  Complexity  of  scenarios 


2.3.1.  Mission  types 

2.3.2.  Mission  segments 


2.3.3.  Number  of  players 


2.3.4.  Number  and  types  of  assets 


2  3.5.  Number  and  levels  of  team  roles 


2.4.  Ability  to  run  pre-packaged  scenano  to  a  defined 
timeline 


2.5.  Ability  to  control  select  vanables  in  real  time 


2.6.  Ability  to  interject  probes  in  real  time 


2.7.  Ability  to  stop/re-start  scenarios 


2.8.  Ability  to  backup/re-run  scenanos 


2.9.  Ability  to  dynamically  modify  scenano  depending 
upon  responses 


2.10.  Comms  among  expenmenters  /  controllers 


3.  Oata  Collection  .  . .  I' 


3. 1 .  Ability  to  capture  physical  actions: 


3.11.  Movement  in  OR 


Adjustability  and  control  over  lighting 
Adjustability  and  control  over  acoustics 


Availability? 

Modifiabilty?  (e  g.  freeze  and  tracking  probes) 


Location,  threat  type,  TG  configuration,  OOTW,  Aid  civil  power, 
etc 

Part  task  breakdown  capability 


Multiple  players  on  single  task,  or  single  player  multiple  roles. 


How  many  and  what  types  en  /  fr  /  neutral  ships,  a/c,  wpns, 
sensors,  etc. 

Can  these  be  vaned? 


Can  these  be  varied? 


Certain  events  at  certain  points  on  time  line,  regardless  of 
player  response. 

Data  to  be  collected  on  that  time  line. 


Ad  hoc  failures  of  Circuit  or  Net,  workstation  to  fail,  wpn  system 
to  fail. 


Freeze  probes 
Tracking  probes 


Yes,  requires  manual  control 


If  x  then  y  e.g.  if  fails  to  respond  to  threat,  re-msert  same  threat  Yes 
later 

If  a  then  b 


Can  observer  /  exper  provide  instructions,  ask  questions  over 
net  to  players  or  observers  or  controllers?  If  not  can  this  be 
added? 


Normal  -  but  also  background  noise  from  AC  -  no  simulated  systems  noise  (eg. 
Weapons) 

Yes-manual  input  of  all  sim  events  and  comms 
Y 


Same  as  OTTC 


No  multiple  players  for  a  single  position,  but  -  3  possible  for  any  back  row  position. 


Yes 


Yes,  but  manual  sim  input  required 
Yes 


e.g  video,  movement  sensors  in  OR  or  on  person. 


None  -  camera  could  be  added,  this  would  be  on  a  separate  time  base 
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3.1.2.  Workstation 

3. 1.2.1.  Keystrokes 

Quad  video,  software  based? 

N,  but  C-SAVRAS  could  be  added. 

3. 1.2.2.  Screen  navigation 

Quad  video,  software  bsed? 

Yes 

3. 1.2.3.  Comm  buttons/pedals 

On  net,  off  net? 

Yes  -  tied  with  video 

3.1.3.  Other  devices 

Lap  tops,  PDAs,  key  pads,  etc. 

Yes 

3.2.  Ability  to  capture  information  exchange 

3.2.1.  Written 

Non  CCS:  formal  (msg  pad,  format),  informal  (post  it  note,  etc). 
Timing,  content,  source,  addressee(s) 

Yes  -same  as  ORTT 

3.2.2.  Verbal 

Network  vs  non  network 

Timing,  content,  source,  addressee(s) 

No  -  unless  on  intercom  and  C-SAVRAS  is  in  place 

3.2.3.  Visual 

Face  to  face,  stateboaids,  non  CCS  info  sources  (e.g.  GCCS, 
plot,  manuals,  charts,  etc) 

Timing,  content,  addressee(s)  i.e.  what  is  source  of  information. 

Yes  -  need  separate  video 

3.3.  Capability  of  recording  contents  of  CCS  Display 

Time  line,  screen  dump  vs  data  content. 

No,  only  through  C-SAVRAS 

3  4.  Ability  to  capture  timing  of  actions  and  other  events 
to  required  accuracy 

Timeline  resolution,  registration  to  other  workstations,  etc. 

Yes,  no  time  delay  - 1  sec  resolution 

35.  Availability  of  observers  as  assessors. 

Instructor  SMEs  to  act  as  observers  and  interpreters  for  real 
time  or  post  hoc  review. 

Yes -as  for  ORTT 

3.6.  Ability  to  capture  data  in  post  analysis  (by  re-running 
scenano) 

Can  responses  to  scenano  be  re-run? 

How  comprehensive  is  this  capability?  Can  tempo  and/or  content 
be  modified  for  each  re-run.  Can  data  sets  be  aggregated? 

No  -  but  limited  workstations  with  limited  data. 

Takes  more  work 

Yes  for  CCS  data 

No 

3  7.  Existence  of  training  MOPs 

Which  OR  positions?  Doc  available. 

Yes 

. :: . . . 

4.1.  Ability  to  store  data  in  a  format  that  is  readily 

Can  data  be  used  in  other  analysis  display  software 

No  would  need  special  software  to  convert  (see  comments  on  History  recording  under 

exported  for  analysis 

applications? 

C-SAVRAS. 

42.  Ability  to  analyse  data 

Existing  or  on  -site  data  analysis  capabilities? 

Yes 

4.3.  Ability  to  display  data 

Existing  or  on  -site  data  display  capabilities? 

Yes 

4.4.  Reliable  data  storage 

Types  of  reliability  problems 

Crash  risk 

Data  formats 

Compatibility  with 

High  -  relies  on  experienced  operator 

4.5.  Access  to  data  after  collection 

Can  data  be  taken  off  -site? 

Yes,  at  warfare  centre 

Security  issues? 

No,  for  exercise  data 
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5.1.  Objective 

Time  and  accuracy  based  metncs  excluding  rating  scales 

5.2.  Subjective 

Subjective  instructor  based  (criteria,  rating  scales,  etc). 

5.3.  Machine-based 

Automatic  implementation  of  objective  measures  above? 

i 

- 1  EnvlroiwwrrtiiKl General FkWity  *  .  ,  ,_.||iaa.  v, .  m  ...  «  1  ”*  - 

-  fKimg  HwSm*  -  mBSm?*  . 1  '  . s .  *  .  *‘1 

i 

6.1.  Interruption  free 

Unbroken  sequences?  Intrusive  observers? 

Yes,  up  to  sustained  ops 

1 

6.2.  Appropriate  illumination 

Colour,  level 

Yes  same  as  Ops  room 

6  3.  Appropriate  sound  levels 

Weapons,  engine  noise,  talk 

No,  but  A/C  noise  present 

1 

6  4  Appropriate  Ship  motion 

Changes  in  direction,  interference  with  movement,  sickness? 

No 

1 

6  5  Appropnate  clothing 

Is  clothing  representative  (hoods,  gloves,  heat,) 

Yes 

6.6.  Layout  appropnate  to  OR  (where  required) 

Lines  of  sight  to  stateboards,  space  for  notes,  etc. 

Yes 

6.7.  Range  of  tasks 

Can  all  tasks  be  simulated?  What  tasks  cannot? 

Yes 

6.8.  Tempo  of  operations 

Same  pace  as  operations? 

Yes 

6.9.  Communication 

e.g.  volume  and  type  of  text  messages,  responsiveness,  sound, 
level  of  radio  comms,  etc. 

Yes 

6.10.  Complexity 

e.g.  multi-tasking 

Yes 

6.11.  Workload 

Is  workload  representative 

Yes 

6.12.  Duration 

Sustained  operations,  time  of  day,  fatigue. 

Better  than  ORTT  -  could  run  for  a  shift  or  more 

6.13.  Manning  levels 

Realistic  for  operation 

Yes 

6.14.  Threat  levels 

Are  they  realistic? 

Yes 

6.15.  Force  configuration  and  role. 

Enemy  and  fnendly. 

Yes 

6.16.  Stress 

Type,  intensity. 

Same  as  ORTT 

6.17.  Mission  (s) 

e.g  OOTW  players.  Multi-national,  Aid  to  civil  power. 

Same  as  ORTT:  Y 

6.18.  Social  dimensions 

Use  of  intact  teams? 

Yes 

6.19.  Other 

Quals  /  expenence  of  participants?  Background  knowledge  of 
mission  pnor  to  start  scenano. 

Same  as  ORTT. 
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C4.  Ops  Room  (Harbourside/at  sea) 


1.1.  Availability  of  facility 

Booking  process  (advance  notice  etc) 

Program  (annual?) 

Sharing  vs  exclusive 

Duration  of  booking  24/7,  etc. 

Highly  dependent  upon  ship  availability,  which  is  unknown  at  present.  But  when  refit 
in  progress  there  may  be  extended  opportunities  to  access  and  us  OR. 

Booking  procedure  unknown. 

Current  programming  unklnown. 

1.2.  Set  up  time  for  facility 

How  much  work  /  time  to  prepare  from  current  status. 

One  hour 

1 .3.  Number  of  CCS  workstations 

All  linked? 

All  positions  linked  as  in  normal  Ops. 

1  4.  OR  Team  simulated 

OP/SUP/WD/ORO/CO 

Specialties  (EW/SONAR/ETC) 

Outside  sources  (bridge,  helo,  etc) 

All  plus  CANEWS,  HMS,  SPS;  including  GMCS  and  CANTASS 

1 .5.  Suitable  system  docs  &  manuals  available? 

Availability:  what,  where,  removable? 

Suitability:  Contents,  Set  up  vs  maintenance. 

1 .6.  Availability  of  Technical/Controller  support  for 

1  6. 1 .  Software/hardware  during  preparation 

Help  with  set  up  of  equipment 

Scenario  development  and  set  up 

Set  data  capture  equipment,  software,  analysis 

Yes 

1.6.2.  Software/hardware  during  testing 

Maintenance  /  repair  of  crashes 

Ad  hoc  changes  to  scenario  (Unplanned  insertion  of  data) 

Yes 

1 6.3.  Control  scenario  play 

Availability  of  controller  staff  as  scenario  players,  war  game 
directors? 

Yes 

1  6.4.  Software/hardware  dunng  analysis 

Data  capture  and  analysis,  downloading,  configunng  for  other 
analysis  software,  etc. 

Limited:  recording.  Can  only  playback  in  CSTC  or  Ops  Room; 

No  time  tags,  have  to  remember  time  of  events. 

There  is  a  trade-off  between  amount  of  information  recorded  and  playback  time,  could 
be  as  little  as  10  mins  for  everything. 

1 .6.5.  Trg  of  HS©  test  personnel  in  technical  issues 

How  are  HS®™  staff  to  be  trained 

Yes,  easier  than  ORTT. 

1 .7.  Availability  of  Personnel  as  test  participants 

1.7.1.  Number 

How  many  can  we  get? 

1 .7.2.  Experience  /  qualifications 

What  range  of  experience  and  qualification  (e.g.  students  only?) 

1.7.3.  Individuals  or  teams 

Will  they  have  worked  together  before? 

1.7.4.  Notice  required 

How  far  ahead  is  booking  required? 

1 .7.5.  Range  of  OR  team  represented 

Different  availability  for  different  levels?  Are  “filler”  personnel 
available  for  part  task  simulation? 

1 .8.  Video  recording  available 
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1 .9.  Audio  recording  available 


1.9.1.  Type  of  audio 


mm 


1.10.  Information  exchange  (is  it  represented? 


1101  Internal  net  in  OR 


1 . 1 0.2.  External  nets  to  OR  on  ship 


1.10.3.  External  nets  to  ship 


1.10.4.  Text 


1.10.5.  Face  to  face  in  OR 


1 .1 0.6.  Other  information  exchanges 


1 11.  Customise  environment  for  data  collection 


1.11.1.  Extra  audio  and  video 


1 .1 1 .2.  Add  on  software  modules 


1.12.  Standalone  keypads,  notebooks  or  PDA's  for  data 
entry  by  participants  or  observers? 


1.13.  Reliability  of  system 


1.14.  Space  for  observers 


#  workstations  covered  (internal  vs  external  to  OR), 
duration  of  recording, 

Time  coord  with  different  video  /  audio  /other  sources, 
identification  of  position(s),  incoming  vs  outgoing 
audio  quality  (noise,  etc) 
off  net  speech  in  OR 


Who  can  be  on  it  in  trainer? 


Who  can  be  on  it  trainer? 


Who  can  be  on  it  trainer? 


What  types  of  text  messages? 
Level  of  text  message  activity? 


Direct  voice  contact  in  OR? 


State  boards 

Manuals  /  Charts  /  Tacpacs  /  Post  its  /  etc. 


Ability  to  add  COMDAT  prototype  equipment  /  software 
Technical  support  for  this? 


Add  extra  audio  /  video  data  capture  devices 


Ability  to  add  on  software  modules  to  inteiject  data  into  canned 
scenanos/simulations 


Eg  for  workload  ratings  at  intervals 
Or  for  response  to  freeze  probes 
or  for  diary  entry 


Types  of  reliability  problems 
Repair  time  /  typical  down  time 
Repeatability  of  scenanos 


Is  there  space  for  more  people  to  observe  /  capture  data. 
Worst  case  =  one  observer  at  each  work  station. 


No,  but  the  capability  is  there.  Passive  audio  tape  could  be  implemented  but  would 
not  be  co-ordinated  with  software  data  capture.  C-SAVRAS  may  also  be  plugged  into 
any  6  ws. 

Off-network  audio  would  require  manual  recording. 

Quality  good 


Yes,  all  inputs  are  put  in  real  time,  cannot  be  pre-programmed 


Yes 


Yes 


Yes,  but  manually  handled 


No-  this  would  need  to  be  done  manually 
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2.1.  Environmental  control 

(lighting,  acoustics,  thermal,  motion) 


2.2.  Pre-packaged  scenario 


2.3.  Complexity  of  scenanos 


2.3.1.  Mission  types 


2.3.2.  Mission  segments 


2.3.3.  Number  of  players 


2.3.4.  Number  and  types  of  assets 


2.3.5.  Number  and  levels  of  team  roles 


2.4.  Ability  to  run  pre-packaged  scenario  to  a  defined 
timeline 


2  5  Ability  to  control  select  vanables  in  real  time 


2.6.  Ability  to  interject  probes  in  real  time 


2.7.  Ability  to  stop/re-start  scenanos 


2.8.  Ability  to  backup/re-run  scenanos 


2.9.  Ability  to  dynamically  modify  scenano 
depending  upon  responses 


2.10.  Comms  among  expenmenters  /  controllers 


Adjustability  and  control  over  lighting 
Adjustability  and  control  over  acoustics 


Availability? 

Modifiabilty?  (e.g.  freeze  and  tracking  probes) 


Location,  threat  type,  TG  configuration,  OOTW,  Aid  civil  power, 
etc 


Part  task  breakdown  capability 


Multiple  players  on  single  task,  or  single  player  multiple  roles. 


How  many  and  what  types  en  /  fr/  neutral  ships,  a/c,  wpns, 
sensors,  etc. 

Can  these  be  varied? 


Can  these  be  varied? 


Certain  events  at  certain  points  on  time  line,  regardless  of 
player  response. 

Data  to  be  collected  on  that  time  line. 


Yes-manual  input  of  all  sim  events  and  comms 
Y 


Same  as  OTTC 


No  multiple  players  fora  single  position,  but  -  3  possible  for  any  back  row  position. 


Yes 


Yes,  but  manual  sim  input  required 


Ad  hoc  failures  of  Circuit  or  Net,  workstation  to  fail,  wpn  system  Yes 
to  fail. 


Freeze  probes 
Tracking  probes 


Yes,  requires  manual  control 


If  x  then  y  e.g.  if  fails  to  respond  to  threat,  re-insert  same  threat  Yes 
later 

If  a  then  b 


Can  observer  /  expr  provide  instructions,  ask  questions  over 
net  to  players  or  observers  or  controllers?  If  not  can  this  be 
added? 


3.1.  Ability  to  capture  physical  actions: 


3.1.1.  Movement  in  OR 


e  g  video,  movement  sensors  in  OR  or  on  person 


None  -  camera  could  be  added,  this  would  be  on  a  separate  time  base 
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3.1.2.  Workstation 

3.1. 2.1.  Keystrokes 

Quad  video,  software  based? 

N,  but  C-SAVRAS  could  be  added. 

3. 1.2.2.  Screen  navigation 

Quad  video,  software  bsed? 

Yes 

3. 1.2.3.  Comm  buttons/pedals 

On  net,  off  net? 

Yes -tied  with  video 

3.1.3.  Other  devices 

Lap  tops,  PDAs,  key  pads,  etc. 

Yes 

3.2.  Ability  to  capture  information  exchange 

3.2.1.  Wntten 

Non  CCS:  formal  (msg  pad,  format),  informal  (post  it  note,  etc). 
Timing,  content,  source,  addressee(s) 

Yes 

3.2.2.  Verbal 

Netwoik  vs  non  netwoik 

Timing,  content,  source,  addressee(s) 

No  -  unless  on  intercom  and  C-SAVRAS  is  in  place 

3  23.  Visual 

Face  to  face,  stateboards,  non  CCS  info  sources  (e.g.  GCCS, 
plot,  manuals,  charts,  etc) 

Timing,  content,  addressee(s)  i.e.  what  is  source  of  information. 

Yes  -  need  separate  video 

3.3.  Capability  of  recording  contents  of  CCS  Display 

Time  line,  screen  dump  vs  data  content. 

No,  only  through  C-SAVRAS 

3  4  Ability  to  capture  timing  of  actions  and  other  events 
to  required  accuracy 

Timeline  resolution,  registration  to  other  workstations,  etc. 

Yes,  no  time  delay  - 1  sec  resolution 

3.5.  Availability  of  observers  as  assessors. 

Instructor  SMEs  to  act  as  observers  and  interpreters  for  real 
time  or  post  hoc  review. 

Yes  -  as  for  ORTT  and  CSTC 

3.6.  Ability  to  capture  data  in  post  analysis  (by  re-running 
scenano) 

Can  responses  to  scenano  be  re-run? 

How  sophisticated  is  this  capability?  Can  tempo  and/or  content 
be  modified  for  each  re-run.  Can  data  sets  be  aggregated? 

No  -  but  limited  workstations  with  limited  data. 

Takes  more  work 

Yes  for  CCS  data 

No 

3.7.  Existence  of  training  MOPs 

Which  OR  positions?  Doc  available. 

Yes 

^  . *  *  tff  ,  , 
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4 1 .  Ability  to  store  data  in  a  format  that  is  readily 

Can  data  be  used  in  other  analysis  display  software 

No  would  need  special  software  to  convert  (see  comments  on  History  recording  under 

exported  for  analysis 

applications? 

C-SAVRAS. 

4.2.  Ability  to  analyse  data 

Existing  or  on  -site  data  analysis  capabilities? 

Yes 

4.3.  Ability  to  display  data 

Existing  or  on  -site  data  display  capabilities? 

Yes 

4.4.  Reliable  data  storage 

Types  of  reliability  problems 

Crash  risk 

Data  formats 

Compatibility  with 

High  -  relies  on  experienced  operator 

Compatible  with  Excel  using  existing  software  conversion  of  specific  data  elements. 

4.5.  Access  to  data  after  collection 

Can  data  be  taken  off  -site? 

Security  issues? 

Yes,  at  warfare  centre 

No,  for  exercise  data 
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51.  Objective 

Time  and  accuracy  based  metrics  excluding  rating  scales 

5.2.  Subjective 

Subjective  instructor  based  (criteria,  rating  scales,  etc). 

5.3.  Machine-based 

Automatic  implementation  of  objective  measures  above? 

6.1.  Interruption  free 

Unbroken  sequences?  Intrusive  observers? 

Yes,  up  to  sustained  ops 

6.2.  Appropriate  illumination 

Colour,  level 

NA 

6.3.  Appropriate  sound  levels 

Weapons,  engine  noise,  talk 

NA 

6.4.  Appropnate  Ship  motion 

Changes  in  direction,  interference  with  movement,  sickness? 

No 

6.5.  Appropriate  clothing 

Is  clothing  representative  (hoods,  gloves,  heat,) 

Yes 

6.6.  Layout  appropriate  to  OR  (where  required) 

Lines  of  sight  to  stateboards,  space  for  notes,  etc. 

Yes 

6.7.  Range  of  tasks 

Can  all  tasks  be  simulated?  What  tasks  cannot? 

Yes 

6.8.  Tempo  of  operations 

Same  pace  as  operations? 

Yes 

6.9.  Communication 

e.g.  volume  and  type  of  text  messages,  responsiveness,  sound, 
level  of  radio  comms,  etc. 

Yes 

6.10.  Complexity 

e.g.  multi-tasking 

Yes 

6.11.  Workload 

Is  workload  representative 

Yes 

612.  Duration 

Sustained  operations,  time  of  day,  fatigue 

Better  than  ORTT  •  could  run  for  a  shift  or  more 

613  Manning  levels 

Realistic  for  operation 

Yes 

6.14.  Threat  levels 

Are  they  realistic? 

Yes 

6.15.  Force  configuration  and  role. 

Enemy  and  friendly. 

Yes 

616.  Stress 

Type,  intensity. 

Same  as  ORTT,  CSTC 

6.17.  Mission  (s) 

e  g.  OOTW  players.  Multi-national,  Aid  to  civil  power. 

Yes 

6.18.  Social  dimensions 

Use  of  intact  teams? 

Yes 

6.19.  Other 

Quals  /  experience  of  participants?  Background  knowledge  of 
mission  prior  to  start  scenario. 

Same  as  ORTT. 
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Annex  D:  Glossary  of  Technical  Terms  and 
Common  Abbreviations 


AAW 

ASUW 

AOP 

ASW 

ASWC 

AWSS 

CANEWS 

CANTASS 

C2 

CCS 

CMOG  1 

CO 

CPF 

C-SAVRAS 

CSTC 

CIWS 

DCIEM 

DREV 

ESM 

EW 

EWS 

EWSS 

FCS 

GCCS 

GO(s) 

HMS 

HR 

HSI 

IS 

L-M 

MD 


Anti-Air  Warfare 
Anti-Surface  Warfare 
Area  of  Probability 
Anti-Submarine  Warfare 

Assistant  Sensor  Weapons  Controller,  also  Anti-submarine  Warfare  Commander 

Acoustic  Warfare  Support  Specialist 

Canadian  Electronic  Warfare  System 

Canadian  Towed  Array  Sonar  System 

Command  and  Control 

Command  and  Control  System 

Commander  Maritime  Operations  Group  One 

Commanding  Officer 

Canadian  Patrol  Frigate 

Combat  System  Audio  Visual  Recording  and  Analysis  System 
Combat  Systems  Training  Centre 
Close  In  Weapon  System 

Defence  and  Civil  Institute  for  Environmental  Medicine 

Defence  Research  Establishment  Valcartier 

Electronic  Support  Measures 

Electronic  Warfare 

Electronic  Warfare  Supervisor 

EW  Support  Specialist 

Fire  Control  Supervisor 

Global  Command  and  Control  System 

Gamepiece  Operators  (s) 

Hull -Mounted  Sonar 
History  Record 
Humansystems  Incorporated 
Instructor  Station 
Lockheed-Martin 
McDonald  Dettweiler 


Humansysfenw  Incorporated 
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MOP(s)  Measure(s)  of  Performance 

MOG  1  Maritime  Operations  Group  One 

MPA  Maritime  Patrol  Aircraft 

MSDF  Multi-Sensor  Data  Fusion 


NAV  COMMS  Naval  Communicators 


NCM 

NCOT 

OCS 

OMI 

OOW 


Non-Commissioned  Member 
Naval  Combat  Operator  Trainer 
Officer  Conducting  the  Serial 
Operator  Machine  Interface 
Officer  of  the  Watch 


Ops  Room 

ORO 

ORS 

ORTT 

OSS 

QAB 

RCP 

RT  (1,2) 

RAMSES 

SAC 

SASIE 

SATCOM 

SCS 

SHINCOM 

SigOp 

SimOps 

SPS 

STIRs 

SWC 

SME(s) 

TAC  AAW 

TAC  AsuW 

TAC  ASW 

TAC  CCIS 


HALIFAX  Class  Operations  Room 
Operations  Room  Officer 
Operations  Room  Supervisor 
Operations  Room  Team  Trainer 
Ownship  Support  Specialist 
Quick-Action  Button 
Remote  Control  Panel 
Radar  Tracker  (1  and  2) 

Reprogrammable  Advanced  Multimode  Shipboard  ECM  System 

Shipboard  Air  Controller 

System  Analysis  Simulation  Evaluation 

Satellite  Communications 

Sonar  Control  Supervisor 

Shipboard  Integrated  Communications  System 

Signals  Operator 

Simulated  Operations  Room 

Sonobuoy  Processing  System 

Separate  Tracking  and  Illuminating  Radars 

Sensor  Weapons  Controller  (also  Surface  Warfare  Commander) 

Subject  Matter  Expert(s) 

Anti-Air  Warfare  Tactical  Instructor 
Anti-Surface  Warfare  Tactical  Instructor 
Anti-Submarine  Warfare  Tactical  Instructor 

Tactical  Command,  Command  and  Control  Information  System  Instructor 
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TCO 

Trainer  Control  Officer 

TCS 

Trainer  Control  Supervisor 

TS 

Track  Supervisor 

T&E 

Test  and  Evaluation 

TTTO 

Tactics  and  Team  Training  Officer 

WCP 

Weapons  Control  Panel 

XBT 

Expendable  Bathythermograph 

XSV 

Expendable  Sound  Velocimeter 
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14.  ABSTRACT 

I 

(U)  This  report  reviews  five  training  simulator  facilities  in  the  Canadian  Navy  for  their  suitability  in 
conducting  test  and  evaluation  (T&E)  in  support  of  the  COMDAT  multi-sensor  data  fusion  (MSDF) 
i  initiative.  The  facilities  are  the  Operations  Room  Team  Trainer  (ORTT),  the  Naval  Combat  Operator 
Trainer  (NCOT),  the  Combat  Systems  Training  Centre  (CSTC)  and  HALIFAX  Class  Ops  Room  at 
harbourside  or  at  sea.  A  set  of  detailed  capabilities  for  conducting  T&E  was  developed  under  the  four 
major  categories  of  :  Logistics,  Experimental  Control,  Data  Collection  and  Data  Storage  and  Analysis. 
Each  of  the  potential  facilities  was  reviewed  against  these  categories  and  the  capabilities  of  each  facility 
were  then  assessed  for  their  suitability  for  conducting  the  expected  range  of  T&E  activities.  It  was 
concluded  that  the  NCOT  would  be  the  most  suitable  facility  for  conducting  T&E  using  single  operators 
and  small  teams  performing  simple  operational  functions  such  as  detecting  and  tracking  radar  contacts. 
For  more  complex  functions  in  the  Ops  Room,  including  tasks  involving  interactions  between  complete 
warfare  teams,  interactions  between  warfare  directors  and  the  ORO,  and  between  the  ORO,  CO  and  the 
entire  Ops  Room  team,  the  ORTT  represents  the  appropriate  environment  for  conducting  T&E  activities. 
A  range  of  scenarios  exists  for  the  ORTT  and  for  NCOT,  which  may  be  adaptable  for  T&E  purposes. 
Severe  constraints  on  the  availability  of  the  ORTT  (and  possibly  the  NCOT)  and  the  associated  personnel 
infrastructure  were  noted,  due  to  existing  training  demand  for  the  Navy.  It  is  recommended  that  a  simple 
proof  of  concept  trial  for  assessing  the  suitability  NCOT  for  T&E  be  conducted  as  soon  as  possible. 
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